Consolidate data from multiple machines
Consolidate data from multiple machines
One of the most common forwarding use cases is to consolidate data originating across numerous machines. Forwarders located on the machines forward the data to a central Splunk indexer. With their small footprint, universal forwarders ordinarily have little impact on their machines' performance. This diagram illustrates a common scenario, where universal forwarders residing on machines running diverse operating systems send data to a single Splunk instance, which indexes and provides search capabilities across all the data:
The diagram illustrates a small deployment. In practice, the number of universal forwarders in a data consolidation use case could number upwards into the thousands.
This type of use case is simple to configure:
1. Determine what data, originating from which machines, you need to access.
2. Install a Splunk instance, typically on its own server. This instance will function as the receiver. All indexing and searching will occur on it.
3. Enable the instance as a receiver through Splunk Web or the CLI. Using the CLI, enter this command from $SPLUNK_HOME/bin/:
./splunk enable listen <port> -auth <username>:<password>
For <port>, substitute the port you want the receiver to listen on. This also known as the "receiver port".
4. If any of the universal forwarders will be running on a different operating system from the receiver, install the app for the forwarder's OS on the receiver. For example, assume the receiver in the diagram above is running on a Linux box. In that case, you'll need to install the Windows app on the receiver. You might need to install the *nix app, as well. -- However, since the receiver is on Linux, you probably have already installed that app. Details and provisos regarding this can be found here.
After you have downloaded the relevant app, remove its inputs.conf file before enabling it, to ensure that its default inputs are not added to your indexer. For the Windows app, the location is: $SPLUNK_HOME/etc/apps/windows/default/inputs.conf.
5. Install universal forwarders on each machine that will be generating data. These will forward the data to the receiver.
6. Set up inputs for each forwarder. See "What Splunk can index".
7. Configure each forwarder to forward data to the receiver. For Windows forwarders, you can do this at installation time, as described here. For *nix forwarders, you must do this through the CLI:
./splunk add forward-server <host>:<port> -auth <username>:<password>
For <host>:<port>, substitute the host and receiver port number of the receiver. For example, splunk_indexer.acme.com:9995.
Alternatively, if you have many forwarders, you can use an outputs.conf file to specify the receiver. For example:
[tcpout:my_indexers] server= splunk_indexer.acme.com:9995
You can create this file once, then distribute copies of it to each forwarder.
This documentation applies to the following versions of Splunk: 4.2 , 4.2.1 , 4.2.2 , 4.2.3 , 4.2.4 , 4.2.5 , 4.3 , 4.3.1 , 4.3.2 , 4.3.3 , 4.3.4 , 4.3.5 , 4.3.6 , 5.0 , 5.0.1 , 5.0.2 View the Article History for its revisions.
Comments
documentation here is extremely vague, for example how can you simply configure splunk universal forwarder to forward data from X log files to the indexer?
There is nothing in here that give you any valid deployment example and how to configure it.
hi Blaise,
there is no need to create a specific user on the receiver, as long as the receiver is configured to accept traffic from your forwarder(s):
http://www.splunk.com/base/Documentation/latest/Deploy/Enableareceiver
this topic is an overview--other topics in this chapter give detailed procedures for configuring forwarders and receivers.
Could you please define where and how the user is created ? do you need to create a specific user on the receiver ? at the OS level or only at splunk level ?
it is not clear and I am struggling to understand how to do this ... thank you
Thanks for the comments, Moonsoft. Information and examples about configuring forwarders are contained in the Getting Data In manual. If you follow the link in step 6 of the procedure in this topic, you can begin to read about that subject. The specific information you are looking for might be contained in this topic: http://www.splunk.com/base/Documentation/latest/Data/Configureyourinputs.