Forwarding and Receiving

Forwarding and Receiving

Any Splunk Server can forward some or all of its incoming data to other Splunk Servers. A server with a valid Splunk Professional license can also receive data from other Splunk Servers.

Don't confuse Forwarding and Receiving with Distributed Search, which only distributes searches rather than the indexed data to be searched.

How to configure receiving

You'll need a Splunk Professional license for the receiving server.

Do not try to receive Splunk-2-Splunk data on the Splunk Server management port (8089). The management port uses an HTTP protocol. Data forwarding and receiving uses TCP. Set up a different port - many customers use port 9997.

Web interface

Go to the Admin section, select the Splunk 2 Splunk tab, and click Receive Data. To begin recieving data:

1) Set the radio button to Yes.

2) Specify the port that you want Splunk to listen on. This port will be the port that all other instances of Splunk that forward data will you use when sending events.

3) Click the Save button to commit the configuration. You will need to restart the server for your changes to take effect.

Command line

To enable listening:

# ./splunk enable listen 42099 -auth admin:changeme
Listening for Splunk data on TCP port 42099.

To disable listening:

# ./splunk disable listen -auth admin:changeme
No longer listening for Splunk TCP data.
You need to restart the Splunk Server for your changes to take effect.

Configuration File

These commands will create an entry in $SPLUNK_HOME/etc/bundles/local/inputs.conf that looks like:

	[splunktcp://42099]
	queue = indexQueue
	disabled = false

How to configure forwarding

Any Splunk Server can be configured to forward its data to any other Splunk Server.

Do not forward to the Splunk Server management port (8089). The management port uses an HTTP protocol. Splunk-2-Splunk data forwarding uses TCP. Set up a different port - many customers use port 9997.

Web interface

Don't configure two Splunk Servers to forward to each other. See instructions for a redundant setup below.

Go to the Admin section, select the Splunk 2 Splunk tab, and click Forward Data. To begin forwarding data:

1) Set the radio button to Yes.

2) Specify the Splunk server(s) and port number that you should be sending Splunk TCP data to. The port number should be the same one that you specified when you configured receiving.

3) Click the Save button to commit the configuration. You will need to restart the server for your changes to take effect.

Command line

Don't configure two Splunk Severs to forward to each other. See instructions for a redundant setup below.

To enable forwarding:

# ./splunk add forward-server 10.2.2.2:9999 -auth admin:changeme
Added Splunk-2-Splunk forwarding to: 10.2.2.2:9999

To disable forwarding:

# ./splunk remove forward-server 10.2.2.2:9999 -auth admin:changeme
Stopped Splunk-2-Splunk forwarding to: 10.2.2.2:9999
You need to restart the Splunk Server for your changes to take effect.

Configuration File

Don't configure two Splunk Severs to forward to each other. See instructions for a redundant setup below.

Forwarding of Splunk data is controlled in $SPLUNK_HOME/etct/modules/output/TCP/config.xml

The section of the XML pertaining to Forwarding is:

    <processor name="outboundTCP-10-10-10-10-port-9999" plugin="tcpoutputprocessor">
         <config>
            <uri>10.10.10.10:9999</uri>
           <maxNumberOfRetriesAtHighestBackoff>0</maxNumberOfRetriesAtHighestBackoff>
            <initialBackoff>1</initialBackoff>
            <maxBackoff>20</maxBackoff>
            <backoffAtStartup>5</backoffAtStartup>
            <usePersistentQueue>true</usePersistentQueue>
            <blockOnQueueFull>false</blockOnQueueFull>
         </config>
    </processor>

In the event that the Splunk instance loses connectivity to the network, Splunk will queue up to 100,000 events to prevent a loss of data. If you need to increase the size of this buffer add the tag <maxQueueSize></maxQueueSize> to the $SPLUNK_HOME/etct/modules/output/TCP/config.xml

SSH tunnel forwarding / receiving

An SSH tunnel can be configured between two Splunk Servers for secure

Splunk-2-Splunk data forwarding and receiving. Both Splunk Servers must

be running sshd.

1. Select unused ports on each Splunk Server. In this example we will use port

7779 on the sending Splunk Server and 9997 on the receiving Splunk Server.

1. Configure the receiving Splunk Server to listen on port 9997 by supplying

that value to the receive data form available in the Splunk-2-Splunk tab in

the Admin section of the Splunk Server GUI.

1. On the sending Splunk Server, execute this command as the user running

Splunk Server (typically root):

ssh -2 -N -f -L 7779:remoteSplunkServerHostName:9997 root@remoteSplunkServerHostName

1. Configure the sending Splunk Server to forward data to 127.0.0.1, port 7779.

Optionally decide if a local copy of the forwarding data is desired.

1. Restart the sending Splunk Server.

Data indexed on the sending Splunk Server is now being securely forwarded to

the receiving Splunk Server via the SSH tunnel. It is generally a good idea

to deploy simple scripts to create, destroy, and monitor the SSH tunnel.

Implementation of these administrative scripts are beyond the scope of this

document.

SSH tips

To avoid having to type in a password when establishing the SSH tunnel, create

a key-pair on the sending Splunk Server and append the public key of the

sending Splunk Server to the authorized_keys file in the .ssh directory

of the user's account hosting the SSH tunnel on the receiving Splunk Server.

One can create a key-pair using the ssh-keygen command. Supply an empty

password for the key file or you'll end up having to type in that password

instead.

Complicating matters in the Splunk-2-Splunk UI is how sending Splunk Servers

are identified on the receiving side. In an SSH tunnel configuration the UI

will report data being received from the local (receiving) host name and

the local (receiving) host's management port. The server name shown under

"This Splunk Server" will be the same one shown under "Receives data from".

Redundant / High Availability configurations

For a redundant, high-availability setup, don't try to forward your servers to each other as this will create an infinte loop resulting in your index quickly consuming all available disk space. Instead configure each Splunk Server to access the same data sources as shown in the Deployment Guide diagram.

• Was this topic useful?
• Post a comment