Enable forwarding and receiving

Enable forwarding and receiving

To enable forwarding and receiving, you configure both a receiver and a forwarder. The receiver is the Splunk instance receiving the data; the forwarder is the Splunk instance forwarding the data. Depending on your needs, you might have multiple receivers or forwarders.

You must first set up the receiver. You can then set up forwarder(s) to send data to that receiver.

Important: The receiver must be running the same (or later) version of Splunk as its forwarder. A 4.0 receiver can receive data from a 3.4 forwarder, but a 3.4 receiver cannot receive from a 4.0 forwarder.

Set up receiving

You enable receiving in Splunk Web or through the Splunk CLI.

Set up receiving with Splunk Web

Use the Manager interface to set up a receiver:

1. Log into Splunk Web as admin on the server that will be receiving data from a forwarder.

2. Click the Manager link in the upper right corner.

3. Select Forwarding and receiving under System configurations.

4. Click Add new in the Receive data section.

5. Specify which TCP port you want the receiver to listen on. For example, if you enter "9997," the receiver will receive data on port 9997. By convention, receivers listen on port 9997, but you can specify any unused port. You can use a tool like netstat to determine what ports are available on your system. Make sure the port you select is not in use by splunkweb or splunkd.

6. Click Save. You must restart Splunk to complete the process.

Set up receiving with Splunk CLI

To access the CLI, first navigate to $SPLUNK_HOME/bin/. This is unnecessary if you have added Splunk to your path.

To enable receiving, enter:

./splunk enable listen <port> -auth <username>:<password>

For <port>, substitute the port you want the receiver to listen on.

To disable receiving, enter:

./splunk disable listen -port <port> -auth <username>:<password>

Searching data received from a forwarder running on a different operating system

In most cases, a Splunk instance receiving data from a forwarder on a different OS will need to install the app for that OS. However, there are numerous subtleties that affect this; read on for the details.

Forwarding and indexing are OS-independent operations. Splunk supports any combination of forwarders and receivers, as long as each is running on a certified OS. For example, a Linux receiver can index data from a Windows forwarder.

Once data has been forwarded and indexed, the next step is to search or perform other knowledge-based activities on the data. At this point, the Splunk instance performing such activities might need information about the OS whose data it is examining. You typically handle this by installing the app specific to that OS. For example, if you want a Linux instance to search OS-specific data forwarded from Windows, you will ordinarily want to install the Windows app on the Linux instance.

If the data you're interested in is not OS-specific, such as web logs, then you do not need to install the Splunk OS app.

In addition, if the receiver is only indexing the data, and an external search head is performing the actual searches, you do not need to install the OS app on the receiver, but you might need to install it on the search head. As an alternative, you can use a search head running the OS. For example, to search data forwarded from Windows to a Linux receiver, you can use a Windows search head pointing to the Linux indexer as a remote search peer. For more information on search heads, see "Set up distributed search".

Important: After you have downloaded the relevant OS app, remove its inputs.conf file before enabling it, to ensure that its default inputs are not added to your indexer. For the Windows app, the location is: %SPLUNK_HOME%\etc\apps\windows\default\inputs.conf.

In summary, you only need to install the app for the forwarder's OS on the receiver (or search head) if it will be performing searches on the forwarded OS data.

Set up forwarding

You can use Splunk Web or the Splunk CLI as a quick way to enable forwarding in a Splunk instance.

You can also enable, as well as configure, forwarding by creating an outputs.conf file for the Splunk instance. Although setting up forwarders with outputs.conf requires a bit more initial knowledge, there are obvious advantages to performing all forwarder configurations in a single location. Most advanced configuration options are available only through outputs.conf. In addition, if you will be enabling and configuring a number of forwarders, you can easily accomplish this by editing a single outputs.conf file and making a copy for each forwarder. See the topic Configure fowarders with outputs.conf for more information.

Note: By default, Splunk uses an Enterprise trial license when it is initially installed. When you enable a forwarder, you should also apply either the forwarder license or the free license to avoid any subsequent license issues. Instructions on how to do this can be found here.

Set up regular forwarding with Splunk Web

Use the Manager interface to set up a forwarder. To set up a regular forwarder:

1. Log into Splunk Web as admin on the server that will be forwarding data.

2. Click the Manager link in the upper right corner.

3. Select Forwarding and receiving under System configurations.

4. Click Add new in the Forward data section.

5. Enter the hostname or IP address for the receiving Splunk instance, along with the port specified when the receiver was configured. For example, you might enter: receivingserver.com:9997.

6. Click Save. You must restart Splunk to complete the process.

You can use Splunk Web to perform one other configuration (for regular forwarders only). To store a copy of indexed data local to the forwarder:

1. From Forwarding and receiving, select Forwarding defaults.

2. Select Yes to store and maintain a local copy of the indexed data on the forwarder.

All other configuration must be done in outputs.conf.

Set up light forwarding with Splunk Web

To enable light forwarding, you must first enable regular forwarding on the Splunk instance. Then you separately enable light fowarding. This procedure combines the two processes:

1. Log into Splunk Web as admin on the server that will be forwarding data.

2. Click the Manager link in the upper right corner.

3. Select Forwarding and receiving under System configurations.

4. Click Add new in the Forward data section.

5. Enter the hostname or IP address for the receiving Splunk instance, along with the port specified when the receiver was configured. For example, you might enter: receivingserver.com:9997.

6. Click Save.

7. Return to Manager>>Forwarding and receiving.

8. Click Enable lightweight forwarding in the Forward data section. You must restart Splunk to complete the process.

Important: When you enable a light forwarder, Splunk Web is immediately disabled. You will then need to use the Splunk CLI or outputs.conf to perform any further configuration on the forwarder. Therefore, if you want to use Splunk Web to configure your forwarder, do so before you enable light forwarding.

Set up forwarding with the Splunk CLI

With the CLI, setting up forwarding is a two step process. First you enable forwarding on the Splunk instance. Then you start forwarding to a specified receiver.

To access the CLI, first navigate to $SPLUNK_HOME/bin/. This is unnecessary if you have added Splunk to your path.

To enable the forwarder mode, enter:

./splunk enable app [SplunkForwarder|SplunkLightForwarder] -auth <username>:<password>

Important: After this step, make sure you restart your Splunk instance as indicated! Attempting to start forwarding activity using the CLI before restarting splunkd will not work!

To disable the forwarder mode, enter:

./splunk disable app [SplunkForwarder|SplunkLightForwarder] -auth <username>:<password>

By disabling forwarding, this command reverts the Splunk instance to a full server.

Start forwarding activity from the Splunk CLI

To access the CLI, first navigate to $SPLUNK_HOME/bin/. This is unnecessary if you have added Splunk to your path.

To start forwarding activity, enter:

./splunk add forward-server <host>:<port> -auth <username>:<password>

To end forwarding activity, enter:

./splunk remove forward-server <host>:<port> -auth <username>:<password>

Note: Although this command ends forwarding activity, the Splunk instance remains configured as a forwarder. To revert the instance to a full Splunk server, use the disable command:

./splunk disable app [SplunkForwarder|SplunkLightForwarder] -auth <username>:<password>

Important: Make sure you restart your Splunk instance as indicated by the CLI to take these changes into account.

Troubleshoot forwarding and receiving

Confusing the receiver's listening and management ports

As part of setting up a fowarder, you specify the receiver (hostname/IP_address and port) that the forwarder will send data to. When you do so, be sure to specify the port that was designated as the receiver's listening port at the time the receiver was configured. See "Set up receiving with Splunk". Do not specify the receiver's management port. If you do mistakenly specify the receiver's management port, the receiver will generate an error similar to this:

splunkd.log:03-01-2010 13:35:28.653 ERROR TcpInputFd - SSL Error = error:140760FC:SSL routines:SSL23_GET_CLIENT_HELLO:unknown protocol
splunkd.log:03-01-2010 13:35:28.653 ERROR TcpInputFd - ACCEPT_RESULT=-1 VERIFY_RESULT=0
splunkd.log:03-01-2010 13:35:28.653 ERROR TcpInputFd - SSL Error for fd from HOST:localhost.localdomain, IP:127.0.0.1, PORT:53075
splunkd.log:03-01-2010 13:35:28.653 ERROR TcpInputFd - SSL Error = error:140760FC:SSL routines:SSL23_GET_CLIENT_HELLO:unknown protocol
splunkd.log:03-01-2010 13:35:28.653 ERROR TcpInputFd - ACCEPT_RESULT=-1 VERIFY_RESULT=0
splunkd.log:03-01-2010 13:35:28.653 ERROR TcpInputFd - SSL Error for fd from HOST:localhost.localdomain, IP:127.0.0.1, PORT:53076
splunkd.log:03-01-2010 13:35:28.653 ERROR TcpInputFd - SSL Error = error:140760FC:SSL routines:SSL23_GET_CLIENT_HELLO:unknown protocol
splunkd.log:03-01-2010 13:35:28.654 ERROR TcpInputFd - ACCEPT_RESULT=-1 VERIFY_RESULT=0
splunkd.log:03-01-2010 13:35:28.654 ERROR TcpInputFd - SSL Error for fd from HOST:localhost.localdomain, IP:127.0.0.1, PORT:53077
splunkd.log:03-01-2010 13:35:28.654 ERROR TcpInputFd - SSL Error = error:140760FC:SSL routines:SSL23_GET_CLIENT_HELLO:unknown protocol
splunkd.log:03-01-2010 13:35:28.654 ERROR TcpInputFd - ACCEPT_RESULT=-1 VERIFY_RESULT=0

Closed receiving socket

If a receiving indexer's queues become full, it will close the receiving socket, to prevent additional forwarders from connecting to it. If a forwarder with load-balancing enabled can no longer forward to that receiver, it will send its data to another indexer on its list. If the forwarder does not employ load-balancing, it will hold the data until the problem is resolved.

The receiving socket will reopen automatically when the queue gets unclogged.

Typically, a receiver gets behind on the dataflow because it can no longer write data due to a full disk or because it is itself attempting to forward data to another forwarder that is not accepting data.

The following warning message will appear in splunkd.log if the socket gets blocked:

Stopping all listening ports. Queues blocked for more than N seconds. 

This message will appear when the socket reopens:

Started listening on tcp ports. Queues unblocked. 

Answers

Have questions? Visit Splunk Answers and see what questions and answers the Splunk community has around configuring forwarding.

• Was this topic useful?
• Post a comment