Splunk® Enterprise

Forwarding Data

Download manual as PDF

This documentation does not apply to the most recent version of Splunk. Click here for the latest version.
Download topic as PDF

Enable a receiver

To enable forwarding and receiving, you configure both a receiver and a forwarder. The receiver is the Splunk Enterprise instance receiving the data; the forwarder sends data to the receiver.

Depending on your needs (for example to enable load balancing), you might have multiple receivers for each forwarder. Conversely, a single receiver usually receives data from many forwarders.

The receiver is either a Splunk Enterprise indexer (the typical case) or another forwarder (referred to as an "intermediate forwarder") configured to receive data from forwarders.

You must set up the receiver first. You can then set up forwarders to send data to that receiver.

Set up receiving

Before enabling a Splunk Enterprise instance (either an indexer or a forwarder) as a receiver, you must install it. You can then enable receiving on the instance through Splunk Web, the CLI, or the inputs.conf configuration file.

Set up receiving with Splunk Web

Use Splunk Web to set up a receiver:

1. Log into Splunk Web as admin on the server that will be receiving data from a forwarder.

2. Click the Settings link at the top of the page.

3. Select Forwarding and receiving in the Data area.

4. Click Add new in the Receive data section.

5. Specify which TCP port you want the receiver to listen on (the listening port, also known as the receiving port). For example, if you enter "9997," the receiver will receive data on port 9997. By convention, receivers listen on port 9997, but you can specify any unused port. You can use a tool like netstat to determine what ports are available on your system. Make sure the port you select is not in use by splunkweb or splunkd.

6. Click Save. You must restart the instance to complete the process.

Set up receiving with Splunk CLI

To enable receiving, run the CLI command:

splunk enable listen <port> -auth <username>:<password>

For <port>, substitute the port you want the receiver to listen on (the receiving port). For example, if you enter "9997," the receiver will receive data on port 9997. By convention, receivers listen on port 9997, but you can specify any unused port. You can use a tool like netstat to determine what ports are available on your system. Make sure the port you select is not in use by splunkweb or splunkd.

The splunk enable listen command creates a [splunktcp] stanza in inputs.conf. For example, if you set the port to "9997", it creates the stanza [splunktcp://9997].

Set up receiving with the configuration file

You can enable receiving on your Splunk Enterprise instance by configuring inputs.conf in $SPLUNK_HOME/etc/system/local. To configure a universal forwarder as an intermediate forwarder (a forwarder that functions also as a receiver), use this method.

To enable receiving, add a [splunktcp] stanza that specifies the receiving port. In this example, the receiving port is 9997:

[splunktcp://9997]
disabled = 0

For further details, refer to the inputs.conf spec file.

Note: The forms [splunktcp://9997] and [splunktcp://:9997] (one colon or two) are semantically equivalent. Use either one.

Searching data received from a forwarder running on a different operating system

In most cases, a Splunk Enterprise instance receiving data from a forwarder on a different OS will need to install the app for that OS. However, there are numerous subtleties that affect this; read on for the details.

Forwarding and indexing are OS-independent operations. You can employ any combination of forwarders and receivers, as long as each is running on a certified OS. For example, a Linux receiver can index data from a Windows universal forwarder.

Once data has been forwarded and indexed, the next step is to search or perform other knowledge-based activities on the data. At this point, the instance performing such activities might need information about the OS whose data it is examining. You typically handle this by installing the app specific to that OS. For example, if you want a Linux instance to search OS-specific data forwarded from Windows, you will ordinarily want to install the Windows app on the Linux instance.

If the data you're interested in is not OS-specific, such as web logs, then you do not need to install the Splunk OS app.

In addition, if the receiver is only indexing the data, and an external search head is performing the actual searches, you do not need to install the OS app on the receiver, but you might need to install it on the search head. As an alternative, you can use a search head running the OS. For example, to search data forwarded from Windows to a Linux receiver, you can use a Windows search head pointing to the Linux indexer as a remote search peer. For more information on search heads, see "About distributed search".

Important: After you have downloaded the relevant OS app, remove its inputs.conf file before enabling the app, to ensure that its default inputs are not added to your indexer. For the Windows app, the location is: %SPLUNK_HOME%\etc\apps\windows\default\inputs.conf.

In summary, you only need to install the app for the forwarder's OS on the receiver (or search head) if it will be performing searches on the forwarded OS data.

Troubleshoot forwarder to receiver connectivity

Confusing the receiver's receiver and management ports

As part of setting up a forwarder, you specify the receiver's hostname/IP_address and port. The forwarder uses these to send data to the receiver. Be sure to specify the port that was designated as the receiving port at the time the receiver was configured. If you mistakenly specify the receiver's management port, the receiver will generate an error similar to this:

splunkd.log:03-01-2010 13:35:28.653 ERROR TcpInputFd - SSL Error = error:140760FC:SSL routines:SSL23_GET_CLIENT_HELLO:unknown protocol
splunkd.log:03-01-2010 13:35:28.653 ERROR TcpInputFd - ACCEPT_RESULT=-1 VERIFY_RESULT=0
splunkd.log:03-01-2010 13:35:28.653 ERROR TcpInputFd - SSL Error for fd from HOST:localhost.localdomain, IP:127.0.0.1, PORT:53075
splunkd.log:03-01-2010 13:35:28.653 ERROR TcpInputFd - SSL Error = error:140760FC:SSL routines:SSL23_GET_CLIENT_HELLO:unknown protocol
splunkd.log:03-01-2010 13:35:28.653 ERROR TcpInputFd - ACCEPT_RESULT=-1 VERIFY_RESULT=0
splunkd.log:03-01-2010 13:35:28.653 ERROR TcpInputFd - SSL Error for fd from HOST:localhost.localdomain, IP:127.0.0.1, PORT:53076
splunkd.log:03-01-2010 13:35:28.653 ERROR TcpInputFd - SSL Error = error:140760FC:SSL routines:SSL23_GET_CLIENT_HELLO:unknown protocol
splunkd.log:03-01-2010 13:35:28.654 ERROR TcpInputFd - ACCEPT_RESULT=-1 VERIFY_RESULT=0
splunkd.log:03-01-2010 13:35:28.654 ERROR TcpInputFd - SSL Error for fd from HOST:localhost.localdomain, IP:127.0.0.1, PORT:53077
splunkd.log:03-01-2010 13:35:28.654 ERROR TcpInputFd - SSL Error = error:140760FC:SSL routines:SSL23_GET_CLIENT_HELLO:unknown protocol
splunkd.log:03-01-2010 13:35:28.654 ERROR TcpInputFd - ACCEPT_RESULT=-1 VERIFY_RESULT=0

Closed receiver socket

If a receiving indexer's queues become full, it will close the receiver socket, to prevent additional forwarders from connecting to it. If a forwarder with load-balancing enabled can no longer forward to that receiver, it will send its data to another indexer on its list. If the forwarder does not employ load-balancing, it will hold the data until the problem is resolved.

The receiver socket will reopen automatically when the queue gets unclogged.

Typically, a receiver gets behind on the data flow because it can no longer write data due to a full disk or because it is itself attempting to forward data to another Splunk Enterprise instance that is not accepting data.

The following warning message will appear in splunkd.log if the socket gets blocked:

Stopping all listening ports. Queues blocked for more than N seconds. 

This message will appear when the socket reopens:

Started listening on tcp ports. Queues unblocked. 

Disable receiving

To disable receiving through the CLI, run the splunk disable listen command:

splunk disable listen -port <port> -auth <username>:<password>

You can also disable receiving by deleting the [splunktcp] stanza from inputs.conf.

Answers

Have questions? Visit Splunk Answers and see what questions and answers the Splunk community has around configuring forwarding.

PREVIOUS
Compatibility between forwarders and indexers
  NEXT
Consolidate data from multiple machines

This documentation applies to the following versions of Splunk® Enterprise: 6.0, 6.0.1, 6.0.2, 6.0.3, 6.0.4, 6.0.5, 6.0.6, 6.0.7, 6.0.8, 6.0.9, 6.0.10, 6.0.11, 6.0.12, 6.0.13, 6.0.14, 6.1, 6.1.1, 6.1.2, 6.1.3, 6.1.4, 6.1.5, 6.1.6, 6.1.7, 6.1.8, 6.1.9, 6.1.10, 6.1.11, 6.1.12, 6.1.13, 6.2.0, 6.2.1, 6.2.2, 6.2.3, 6.2.4, 6.2.5, 6.2.6, 6.2.7, 6.2.8, 6.2.9, 6.2.10, 6.2.11, 6.2.12, 6.2.13


Was this documentation topic helpful?

Enter your email address, and someone from the documentation team will respond to you:

Please provide your comments here. Ask a question or make a suggestion.

You must be logged into splunk.com in order to post comments. Log in now.

Please try to keep this discussion focused on the content covered in this documentation topic. If you have a more general question about Splunk functionality or are experiencing a difficulty with Splunk, consider posting a question to Splunkbase Answers.

0 out of 1000 Characters