Connect forwarders directly to peer nodes
These are the main steps for setting up connections between forwarders and peer nodes, using the traditional method of connecting each forwarder directly to each peer node:
1. Configure the peer nodes to receive data from forwarders.
2. Configure the forwarders to send data to the peer nodes.
3. Enable indexer acknowledgment for each forwarder. This step is required to ensure end-to-end data fidelity. If that is not a requirement for your deployment, you can skip this step.
Once you finish setting up the connection, you must configure the data inputs on the forwarders. See "Configure the forwarder's data inputs".
1. Configure the peer nodes to receive data from forwarders
In order for a peer to receive data from forwarders, you must configure the peer's receiving port. For information on how to configure the receiving port, read "Enable a receiver" in the Forwarding Data manual.
One way to specify the receiving port is to edit the peer's inputs.conf file. You can simplify peer input configuration by deploying a single, identical
inputs.conf file across all the peers. The receiving port that you specify in the common copy of
inputs.conf will supersede any ports you enable on each individual peer. For details on how to create and deploy a common
inputs.conf across all peers, read "Update common peer configurations".
2. Configure the forwarders to send data to the peer nodes
When you set up a forwarder, you specify its receiving peer by providing the peer's IP address and receving port number. For example:
10.10.10.1:9997. You do this in the forwarder's outputs.conf file, as described in "Configure forwarders with outputs.conf" in the Forwarding Data manual. To specify the receiving peer, set the
server attribute, like this:
The receiving port that you specify here is the port that you configured on the peer in step 1.
To set up the forwarder to use load-balancing, so that the data goes to multiple peer nodes in sequence, you configure a load-balanced group of receiving peers. For example, this attribute/value pair in
outputs.conf specifies a load-balanced group of three peers:
To learn more about configuring load balancing, read "Set up load balancing" in the Forwarding Data manual.
Note: There are several other ways that you can specify a forwarder's receiving peer(s). For example:
- You can specify the receiving peer during universal forwarder deployment (for Windows universal forwarders only), as described in Install a Windows universal forwarder in the Universal Forwarder manual.
- You can specify the receiver with the CLI command
add forward-server, as described in Enable a receiver in the Forwarding Data manual.
Both of these methods work by modifying the underlying
outputs.conf file. No matter what method you use to specify the receiving peers, you still need to directly edit the underlying
outputs.conf file if you want to turn on indexer acknowledgment, as described in the next step.
3. Enable indexer acknowledgment for each forwarder
This step is required to ensure end-to-end data fidelity. If that is not a requirement for your deployment, you can skip this step.
To ensure that the cluster receives and indexes all incoming data, you must turn on indexer acknowledgment for each forwarder.
Caution: Indexer acknowledgement can, under some circumstances, result in duplicate events. To learn about this issue and how to work around it, see Protect against loss of in-flight data in the Forwarding Data manual.
To configure indexer acknowledgment, set the
useACK attribute in each forwarder's
For detailed information on configuring indexer acknowledgment, read Protect against loss of in-flight data in the Forwarding Data manual.
Caution: For indexer acknowledgment to work properly, the forwarders' wait queues must be configured to the optimal size. For forwarders at version 5.0.4 or above, the system handles this automatically. For earlier version forwarders, follow the instructions in the version of the Protect against loss of in-flight data topic for that forwarder version. Specifically, read the subtopic on adjusting the
Example: A load-balancing forwarder with indexer acknowledgment
Here is a sample
outputs.conf configuration for a forwarder that is using load balancing to send data in sequence to three peers in a cluster. It assumes that each of the peers has previously been configured to use 9997 for its receiving port:
[tcpout] defaultGroup=my_LB_peers [tcpout:my_LB_peers] autoLBFrequency=40 server=10.10.10.1:9997,10.10.10.2:9997,10.10.10.3:9997 useACK=true
The forwarder starts by sending data to one of the peers listed for the
server attribute. After 40 seconds, it switches to another peer, and so on. If, at any time, it doesn't receive acknowledgment from the current receiving node, it resends the data, this time to the next available node.
Use indexer discovery to connect forwarders to peer nodes
Indexer cluster configuration overview
This documentation applies to the following versions of Splunk® Enterprise: 8.2.6, 8.2.7, 8.2.8, 8.2.9, 8.2.10, 8.2.11, 8.2.12, 9.0.0, 9.0.1, 9.0.2, 9.0.3, 9.0.4, 9.0.5, 9.0.6, 9.1.0, 9.1.1