Consolidate data from multiple machines

Consolidate data from multiple machines

One of the most common forwarding use cases is to consolidate data produced across numerous machines. Light forwarders running on machines generating data forward the data to a central Splunk indexer. Such forwarders ordinarily have little impact on their machines' performance. This diagram illustrates a common scenario, where light forwarders residing on machines running diverse operating systems send data to a single Splunk instance, which indexes and provides search capabilities across all the data:

The diagram illustrates a small deployment. In practice, the number of light forwarders in a data consolidation use case could number upwards into the thousands.

This type of use case is simple to configure:

1. Determine what data, originating from which machines, you need to access.

2. Install a Splunk instance, typically on its own server. This instance will function as the receiver. All indexing and searching will occur on it.

3. Enable the receiver through Splunk Web or the CLI. Using the CLI, enter this command from $SPLUNK_HOME/bin/:

./splunk enable listen <port> -auth <username>:<password>

For <port>, substitute the port you want the receiver to listen on.

4. If any of the forwarders will be running on a different operating system from the receiver, install the app for the forwarder's OS on the receiver. For example, assume the receiver in the diagram above is running on a Linux box. In that case, you'll need to install the Windows app on the receiver. You don't need to install the *NIX app. Since the receiver is on Linux, that app was already installed along with the rest of the Splunk instance.

After you have downloaded the relevant app, remove its inputs.conf file before enabling it, to ensure that its default inputs are not added to your indexer. For the Windows app, the location is: $SPLUNK_HOME/etc/apps/windows/default/inputs.conf.

5. Install a Splunk instance on each machine that will be generating data. These will become light forwarders that forward the data to the receiver.

6. Set up inputs for each forwarder. See Add data and configure inputs in this manual.

7. Configure each forwarder through Splunk Web or the CLI. Using the CLI from $SPLUNK_HOME/bin/, first enable each Splunk instance as a light forwarder:

./splunk enable app SplunkLightForwarder -auth <username>:<password>

Next, begin forwarding to the designated receiver:

./splunk add forward-server <host>:<port> -auth <username>:<password>

For <host>:<port>, substitute the host and port number of the receiver. For example, splunk_indexer.acme.com:9995.

Alternatively, if you have many forwarders, you can use an outputs.conf file to specify the receiver. For example:

[tcpout:my_indexers]
server= splunk_indexer.acme.com:9995

You can create this file once, then distribute copies of it to the $SPLUNK_HOME/etc/system/local/ location of each forwarder.

• Was this topic useful?
• Post a comment