Set up load balancing

Set up load balancing

In load balancing, a Splunk forwarder distributes data across several receiving Splunk instances. Each receiver gets a portion of the total data, and together the receivers hold all the data. To access the full set of forwarded data, you will need to set up distributed searching across all the receivers. For information on distributed search, see What is distributed search? in this manual.

Load balancing enables horizontal scaling for improved performance. In addition, its automatic switchover capability ensures resiliency in the face of machine outages. If a machine goes down, the forwarder simply begins sending data to the next available receiver.

Load balancing can also be of use when monitoring data from network devices like routers. To handle syslog and other data generated across port 514, a single forwarder can monitor port 514 and distribute the incoming data across several Splunk indexers.

Splunk forwarders can perform two types of load balancing:

• Automatic load balancing: Forwarder routes data to different servers based on a specified time interval, for example, switching the data stream every 30 seconds, from server A to server B to server C and then back to server A.
• Round-robin load balancing: Forwarder routes data to different servers, switching with each new event, for example, event 1 goes to server A, event 2 to server B, event 3 to server C, and event 4 back to server A.

Regular forwarders can perform both types of load balancing. Light forwarders can perform only automatic load balancing.

For most purposes, automatic load balancing is recommended. It provides greater resiliency if a forwarder or receiver goes down. It also provides greater flexibility and easier configuration, because you can combine it with a DNS list. Round-robin load balancing can result in somewhat more even load balancing, because the forwarder switches receivers with each new event, but, in practice, any advantage diminishes at greater data volumes. In addition, round-robin requires that the forwarder perform parsing, consuming more RAM and CPU without improving the overall resiliency of the system. It is recommended only when you intend to distribute pre-indexing activity out to the edge network.

This diagram shows a distributed search scenario, in which three light forwarders are performing load balancing across three receivers:

Targets for automatic load balancing

When configuring the set of target receivers, you can employ either DNS or static lists.

DNS lists provide greater flexibility and simplified scale-up, particularly for large deployments. Through DNS, you can change the set of receivers without needing to re-edit each forwarder's outputs.conf file.

The main advantage of a static list is that it allows you to specify a different port for each receiver. This is useful if you need to perform load balancing across multiple receivers running on a single host. Each receiver can listen on a separate port.

Static list target

To use a static list for the target, you simply specify each of the receivers in the target group's [tcpout] stanza in the forwarder's outputs.conf file. In this example, the target group consists of three receivers, specified by IP address and port number:

[tcpout: my_LB_indexers]
autoLB=true
server=10.10.10.1:9997,10.10.10.2:9996,10.10.10.3:9995

The forwarder will load balance between the three receivers listed. If one receiver goes down, the forwarder will automatically switch to the next one available.

DNS list target

To use a DNS list, edit your forwarder's outputs.conf file to specify a single host in the target group's [tcpout] stanza. For example:

[tcpout:my_LB_indexers]
autoLB=true
server=splunkreceiver.mycompany.com:9997

In your DNS server, create a DNS A record for each host's IP address, referencing the server name you specified in outputs.conf. For example:

splunkreceiver.mycompany.com   A   10.10.10.1
splunkreceiver.mycompany.com   A   10.10.10.2
splunkreceiver.mycompany.com   A   10.10.10.3

Note: Windows users should use the DNS snap-in against an Active Directory domain controller to add the appropriate A records to DNS.

The Splunk forwarder will use the DNS list to load balance, sending data in intervals, first to 10.10.10.1, then to 10.10.10.2, then to 10.10.10.3, and then to 10.10.10.1 again. If a receiver is not available, the forwarder skips it and sends data to the next one on the list.

If you have a topology with many forwarders, the DNS list method allows you to update the set of receivers by making changes in just a single location, without touching the forwarders' outputs.conf files.

Configure automatic load balancing for horizontal scaling

To configure automatic load balancing, first determine your needs, particularly your horizontal scaling and failover requirements. Then develop a topology based on those needs, possibly including multiple forwarders as well as receivers and a search head to search across the receivers.

Assuming the topology of three forwarders and three receivers illustrated by the diagram at the start of this topic, set up automatic load balancing with these steps:

1. Install and enable a set of three Splunk instances as receivers. This example uses a DNS list to designate the receivers, so they must all listen on the same port. For example, if the port is 9997, enable each receiver by going to its $SPLUNK_HOME/bin/ location and using this CLI command:

./splunk enable listen 9997 -auth <username>:<password>

2. Install and enable the set of light forwarders. Once you've installed these Splunk instances, use this CLI command on each of them to enable forwarding:

./splunk enable app SplunkLightForwarder -auth <username>:<password>

3. Set up a DNS list with an A record for each receiver's IP address:

splunkreceiver.mycompany.com   A   10.10.10.1
splunkreceiver.mycompany.com   A   10.10.10.2
splunkreceiver.mycompany.com   A   10.10.10.3

4. Create a single outputs.conf file for use by all the forwarders. This one specifies the DNS server name used in the DNS list and the port the receivers are listening on:

[tcpout]
indexAndForward=false

[tcpout:my_LB_indexers]
disabled=false
autoLB=true
autoLBFrequency=40
server=splunkreceiver.mycompany.com:9997

This outputs.conf file also uses the autoLB attribute to specify automatic (instead of round-robin) load balancing and the autoLBFrequency attribute to set a frequency of 40 seconds. Every 40 seconds, the forwarders will switch to the next receiver. The default frequency, which rarely needs changing, is 30 seconds.

5. Distribute the outputs.conf file to all the forwarders, placing it in each forwarder's $SPLUNK_HOME/etc/system/local/ directory.

Specify automatic load balancing from the CLI

You can also use the CLI to specify automatic load balancing. You do this when you start forwarding activity to a set of receivers, using this syntax:

./splunk add forward-server <host>:<port> -method autobalance

where <host>:<port> is the host and port number of the receiver.

This example creates a load-balanced group of four receivers:

./splunk add forward-server indexer1:9991 -method autobalance
./splunk add forward-server indexer2:9991 -method autobalance 
./splunk add forward-server indexer3:9991 -method autobalance
./splunk add forward-server indexer4:9991 -method autobalance

• Was this topic useful?
• Post a comment