Distributed Deployment Manual

 


Use the forwarder to create deployment topologies

Set up load balancing

NOTE - Splunk version 4.x reached its End of Life on October 1, 2013. Please see the migration information.

This documentation does not apply to the most recent version of Splunk. Click here for the latest version.

Set up load balancing

With load balancing, a Splunk forwarder distributes data across several receiving Splunk instances. Each receiver gets a portion of the total data, and together the receivers hold all the data. To access the full set of forwarded data, you need to set up distributed searching across all the receivers. For information on distributed search, see "About distributed search" in this manual.

Load balancing enables horizontal scaling for improved performance. In addition, its automatic switchover capability ensures resiliency in the face of machine outages. If a machine goes down, the forwarder simply begins sending data to the next available receiver.

Load balancing can also be of use when getting data from network devices like routers. To handle syslog and other data generated across port 514, a single heavy forwarder can monitor port 514 and distribute the incoming data across several Splunk indexers.

Note: When implementing load balancing between forwarders and receivers, you must use the forwarder's inherent capability. Do not use an external load balancer. The use of external load balancers between forwarders and receivers will not work properly.

How load balancing works

Splunk forwarders perform "automatic load balancing". The forwarder routes data to different indexers based on a specified time interval. For example, assume you have a load-balanced group consisting of three indexers: A, B, and C. At some specified interval, such as every 30 seconds, the forwarder switches the data stream to another indexer in the group, selected at random. So, the forwarder might switch from indexer B to indexer A to indexer C, and so on. If one indexer is down, the forwarder immediately switches to another.

To expand on this a bit, there is a data stream for each of the inputs that the forwarder is configured to monitor. The forwarder determines if it is safe for a data stream to switch to another indexer. Then, at the specified interval, it switches the data stream to the newly selected indexer. If it cannot switch the data stream to the new indexer safely, it keeps the connection to the previous indexer open and continues to send the data stream until it has been safely sent.

Important: Universal forwarders are not able to switch indexers when monitoring TCP network streams of data (including Syslog) unless an EOF is reached or an indexer goes down, at which point the forwarder will switch to the next indexer in the list. Because the universal forwarder does not parse the data and identify event boundaries before forwarding the data to the indexer (unlike a heavy forwarder), it has no way of knowing when it's safe to switch to the next indexer unless it receives an EOF.

Note: Round-robin load balancing, which was previously available as an alternative to automatic load balancing, was deprecated in Splunk version 4.2.

This diagram shows a distributed search scenario, in which three forwarders are performing load balancing across three receivers:

30 admin13 forwardreceive-dsearch.jpg

Targets for load balancing

When configuring the set of target receivers, you can employ either DNS or static lists.

DNS lists provide greater flexibility and simplified scale-up, particularly for large deployments. Through DNS, you can change the set of receivers without needing to re-edit each forwarder's outputs.conf file.

The main advantage of a static list is that it allows you to specify a different port for each receiver. This is useful if you need to perform load balancing across multiple receivers running on a single host. Each receiver can listen on a separate port.

Static list target

To use a static list for the target, you simply specify each of the receivers in the target group's [tcpout] stanza in the forwarder's outputs.conf file. In this example, the target group consists of three receivers, specified by IP address and receiver port number:

[tcpout: my_LB_indexers]
server=10.10.10.1:9997,10.10.10.2:9996,10.10.10.3:9995

The universal forwarder will load balance between the three receivers listed. If one receiver goes down, the forwarder automatically switches to another one on the list.

DNS list target

To use a DNS list, edit your forwarder's outputs.conf file to specify a single host in the target group's [tcpout] stanza. For example:

[tcpout:my_LB_indexers]
server=splunkreceiver.mycompany.com:9997

In your DNS server, create a DNS A record for each host's IP address, referencing the server name you specified in outputs.conf. For example:

splunkreceiver.mycompany.com   A   10.10.10.1
splunkreceiver.mycompany.com   A   10.10.10.2
splunkreceiver.mycompany.com   A   10.10.10.3

The forwarder will use the DNS list to load balance, sending data in intervals, switching among the receivers specified. If a receiver is not available, the forwarder skips it and sends data to another one on the list.

If you have a topology with many forwarders, the DNS list method allows you to update the set of receivers by making changes in just a single location, without touching the forwarders' outputs.conf files.

Configure load balancing for horizontal scaling

To configure load balancing, first determine your needs, particularly your horizontal scaling and failover requirements. Then develop a topology based on those needs, possibly including multiple forwarders, as well as receivers and a search head to search across the receivers.

Assuming the topology of three universal forwarders and three receivers illustrated by the diagram at the start of this topic, set up load balancing with these steps:

1. Install and enable a set of three Splunk instances as receivers. This example uses a DNS list to designate the receivers, so they must all listen on the same port. For example, if the port is 9997, enable each receiver by going to its $SPLUNK_HOME/bin/ location and using this CLI command:

./splunk enable listen 9997 -auth <username>:<password>

2. Install the set of universal forwarders, as described here.

3. Set up a DNS list with an A record for each receiver's IP address:

splunkreceiver.mycompany.com   A   10.10.10.1
splunkreceiver.mycompany.com   A   10.10.10.2
splunkreceiver.mycompany.com   A   10.10.10.3

4. Create a single outputs.conf file for use by all the forwarders. This one specifies the DNS server name used in the DNS list and the port the receivers are listening on:

[tcpout]
defaultGroup=my_LB_indexers

[tcpout:my_LB_indexers]
disabled=false
autoLBFrequency=40
server=splunkreceiver.mycompany.com:9997

This outputs.conf file uses the autoLBFrequency attribute to set a load-balance frequency of 40 seconds. Every 40 seconds, the forwarders will switch to another receiver. The default frequency, which rarely needs changing, is 30 seconds.

5. Distribute the outputs.conf file to all the forwarders. You can use the deployment server to handle the distribution.

Specify load balancing from the CLI

You can also use the CLI to specify load balancing. You do this when you start forwarding activity to a set of receivers, using this syntax:

./splunk add forward-server <host>:<port> -method autobalance

where <host>:<port> is the host and receiver port of the receiver.

This example creates a load-balanced group of four receivers:

./splunk add forward-server indexer1:9997 -method autobalance
./splunk add forward-server indexer2:9997 -method autobalance
./splunk add forward-server indexer3:9997 -method autobalance
./splunk add forward-server indexer4:9997 -method autobalance

This documentation applies to the following versions of Splunk: 4.2 , 4.2.1 , 4.2.2 , 4.2.3 , 4.2.4 , 4.3 , 4.3.1 , 4.3.2 , 4.3.3 , 4.3.4 , 4.3.5 , 4.3.6 , 4.3.7 , 5.0 , 5.0.1 , 5.0.2 , 5.0.3 , 5.0.4 , 5.0.5 , 5.0.6 , 5.0.7 , 5.0.8 , 5.0.9 , 5.0.10 View the Article History for its revisions.


You must be logged into splunk.com in order to post comments. Log in now.

Was this documentation topic helpful?

If you'd like to hear back from us, please provide your email address:

We'd love to hear what you think about this topic or the documentation as a whole. Feedback you enter here will be delivered to the documentation team.

Feedback submitted, thanks!