Splunk® Enterprise

Distributed Deployment Manual

Download manual as PDF

Splunk version 4.x reached its End of Life on October 1, 2013. Please see the migration information.
This documentation does not apply to the most recent version of Splunk. Click here for the latest version.
Download topic as PDF

Consolidate data from multiple machines

One of the most common forwarding use cases is to consolidate data originating across numerous machines. Forwarders located on the machines forward the data to a central Splunk indexer. With their small footprint, universal forwarders ordinarily have little impact on their machines' performance. This diagram illustrates a common scenario, where universal forwarders residing on machines running diverse operating systems send data to a single Splunk instance, which indexes and provides search capabilities across all the data:

30 admin13 forwardreceive-dataforward.jpg

The diagram illustrates a small deployment. In practice, the number of universal forwarders in a data consolidation use case could number upwards into the thousands.

This type of use case is simple to configure:

1. Determine what data, originating from which machines, you need to access.

2. Install a Splunk instance, typically on its own server. This instance will function as the receiver. All indexing and searching will occur on it.

3. Enable the instance as a receiver through Splunk Web or the CLI. Using the CLI, enter this command from $SPLUNK_HOME/bin/:

./splunk enable listen <port> -auth <username>:<password>

For <port>, substitute the port you want the receiver to listen on. This also known as the "receiver port".

4. If any of the universal forwarders will be running on a different operating system from the receiver, install the app for the forwarder's OS on the receiver. For example, assume the receiver in the diagram above is running on a Linux box. In that case, you'll need to install the Windows app on the receiver. You might need to install the *nix app, as well. -- However, since the receiver is on Linux, you probably have already installed that app. Details and provisos regarding this can be found here.

After you have downloaded the relevant app, remove its inputs.conf file before enabling it, to ensure that its default inputs are not added to your indexer. For the Windows app, the location is: $SPLUNK_HOME/etc/apps/windows/default/inputs.conf.

5. Install universal forwarders on each machine that will be generating data. These will forward the data to the receiver.

6. Set up inputs for each forwarder. See "What Splunk can index".

7. Configure each forwarder to forward data to the receiver. For Windows forwarders, you can do this at installation time, as described here. For *nix forwarders, you must do this through the CLI:

./splunk add forward-server <host>:<port> -auth <username>:<password>

For <host>:<port>, substitute the host and receiver port number of the receiver. For example, splunk_indexer.acme.com:9995.

Alternatively, if you have many forwarders, you can use an outputs.conf file to specify the receiver. For example:

[tcpout:my_indexers]
server= splunk_indexer.acme.com:9995

You can create this file once, then distribute copies of it to each forwarder.

PREVIOUS
Protect against loss of in-flight data
  NEXT
Set up load balancing

This documentation applies to the following versions of Splunk® Enterprise: 4.3, 4.3.1, 4.3.2, 4.3.3, 4.3.4, 4.3.5, 4.3.6, 4.3.7, 5.0, 5.0.1, 5.0.2, 5.0.3, 5.0.4, 5.0.5, 5.0.6, 5.0.7, 5.0.8, 5.0.9, 5.0.10, 5.0.11, 5.0.12, 5.0.13, 5.0.14, 5.0.15, 5.0.16, 5.0.17, 5.0.18


Comments

Thanks for the comments, Moonsoft. Information and examples about configuring forwarders are contained in the Getting Data In manual. If you follow the link in step 6 of the procedure in this topic, you can begin to read about that subject. The specific information you are looking for might be contained in this topic: http://www.splunk.com/base/Documentation/latest/Data/Configureyourinputs.

Cgales splunk
July 12, 2011

documentation here is extremely vague, for example how can you simply configure splunk universal forwarder to forward data from X log files to the indexer?<br /><br />There is nothing in here that give you any valid deployment example and how to configure it.

Moonsoft
July 11, 2011

hi Blaise,<br /><br />there is no need to create a specific user on the receiver, as long as the receiver is configured to accept traffic from your forwarder(s): <br />http://www.splunk.com/base/Documentation/latest/Deploy/Enableareceiver<br /><br />this topic is an overview--other topics in this chapter give detailed procedures for configuring forwarders and receivers.

Rachel
June 14, 2011

Could you please define where and how the user is created ? do you need to create a specific user on the receiver ? at the OS level or only at splunk level ?<br />it is not clear and I am struggling to understand how to do this ... thank you

Blaise
June 13, 2011

Was this documentation topic helpful?

Enter your email address, and someone from the documentation team will respond to you:

Please provide your comments here. Ask a question or make a suggestion.

You must be logged into splunk.com in order to post comments. Log in now.

Please try to keep this discussion focused on the content covered in this documentation topic. If you have a more general question about Splunk functionality or are experiencing a difficulty with Splunk, consider posting a question to Splunkbase Answers.

0 out of 1000 Characters