Splunk® Enterprise

Distributed Deployment Manual

Download manual as PDF

Splunk version 4.x reached its End of Life on October 1, 2013. Please see the migration information.
This documentation does not apply to the most recent version of Splunk. Click here for the latest version.
Download topic as PDF

Enable a receiver

To enable forwarding and receiving, you configure both a receiver and a forwarder. The receiver is the Splunk instance receiving the data; the forwarder sends data to the receiver.

Depending on your needs (for example to enable load balancing), you might have multiple receivers for each forwarder. Conversely, a single receiver usually receives data from many forwarders.

The receiver is either a Splunk indexer (the typical case) or another forwarder (referred to as an "intermediate forwarder") configured to receive data from forwarders.

You must set up the receiver first. You can then set up forwarders to send data to that receiver.

Compatibility between forwarders and indexers

These are the compatibility restrictions between versions of forwarders and indexers:

  • 5.x forwarders (universal/light/heavy) are backwards compatible down to 5.0.x indexers.
  • 5.x indexers are backwards compatible with forwarders down to 5.0.x.

For each app, check Splunk Apps for version compatibility.

As a best practice, it is recommended that indexers be at the same or higher version level than the forwarders they're receiving data from.

Set up receiving

Before enabling a Splunk instance (either an indexer or a forwarder) as a receiver you must, of course, first install it.

You can then enable receiving on a Splunk instance through Splunk Web, the CLI, or the inputs.conf configuration file.

Set up receiving with Splunk Web

Use Splunk Web to set up a receiver:

1. Log into Splunk Web as admin on the server that will be receiving data from a forwarder.

2. Click the Manager link at the top of the page.

3. Select Forwarding and receiving in the Data area.

4. Click Add new in the Receive data section.

5. Specify which TCP port you want the receiver to listen on (the listening port, also known as the receiving port). For example, if you enter "9997," the receiver will receive data on port 9997. By convention, receivers listen on port 9997, but you can specify any unused port. You can use a tool like netstat to determine what ports are available on your system. Make sure the port you select is not in use by splunkweb or splunkd.

6. Click Save. You must restart Splunk to complete the process.

Set up receiving with Splunk CLI

To access the CLI, first navigate to $SPLUNK_HOME/bin/. This is unnecessary if you have added Splunk to your path.

To enable receiving, enter:

./splunk enable listen <port> -auth <username>:<password>

For <port>, substitute the port you want the receiver to listen on (the receiving port). For example, if you enter "9997," the receiver will receive data on port 9997. By convention, receivers listen on port 9997, but you can specify any unused port. You can use a tool like netstat to determine what ports are available on your system. Make sure the port you select is not in use by splunkweb or splunkd.

To disable receiving, enter:

./splunk disable listen -port <port> -auth <username>:<password>

Set up receiving with the configuration file

You can enable receiving on your Splunk instance by configuring inputs.conf in $SPLUNK_HOME/etc/system/local. To configure a universal forwarder as an intermediate forwarder (a forwarder that functions also as a receiver), use this method.

To enable receiving, add a [splunktcp] stanza that specifies the receiving port. In this example, the receiving port is 9997:

[splunktcp://9997]
disabled = 0

For further details, refer to the inputs.conf spec file.

Note: The forms [splunktcp://9997] and [splunktcp://:9997] (one colon or two) are semantically equivalent. Use either one.

Searching data received from a forwarder running on a different operating system

In most cases, a Splunk instance receiving data from a forwarder on a different OS will need to install the app for that OS. However, there are numerous subtleties that affect this; read on for the details.

Forwarding and indexing are OS-independent operations. Splunk supports any combination of forwarders and receivers, as long as each is running on a certified OS. For example, a Linux receiver can index data from a Windows universal forwarder.

Once data has been forwarded and indexed, the next step is to search or perform other knowledge-based activities on the data. At this point, the Splunk instance performing such activities might need information about the OS whose data it is examining. You typically handle this by installing the app specific to that OS. For example, if you want a Linux Splunk instance to search OS-specific data forwarded from Windows, you will ordinarily want to install the Windows app on the Linux instance.

If the data you're interested in is not OS-specific, such as web logs, then you do not need to install the Splunk OS app.

In addition, if the receiver is only indexing the data, and an external search head is performing the actual searches, you do not need to install the OS app on the receiver, but you might need to install it on the search head. As an alternative, you can use a search head running the OS. For example, to search data forwarded from Windows to a Linux receiver, you can use a Windows search head pointing to the Linux indexer as a remote search peer. For more information on search heads, see "Set up distributed search".

Important: After you have downloaded the relevant OS app, remove its inputs.conf file before enabling the app, to ensure that its default inputs are not added to your indexer. For the Windows app, the location is: %SPLUNK_HOME%\etc\apps\windows\default\inputs.conf.

In summary, you only need to install the app for the forwarder's OS on the receiver (or search head) if it will be performing searches on the forwarded OS data.

Troubleshoot forwarder to receiver connectivity

Confusing the receiver's receiver and management ports

As part of setting up a forwarder, you specify the receiver's hostname/IP_address and port. The forwarder uses these to send data to the receiver. Be sure to specify the port that was designated as the receiving port at the time the receiver was configured. If you mistakenly specify the receiver's management port, the receiver will generate an error similar to this:

splunkd.log:03-01-2010 13:35:28.653 ERROR TcpInputFd - SSL Error = error:140760FC:SSL routines:SSL23_GET_CLIENT_HELLO:unknown protocol
splunkd.log:03-01-2010 13:35:28.653 ERROR TcpInputFd - ACCEPT_RESULT=-1 VERIFY_RESULT=0
splunkd.log:03-01-2010 13:35:28.653 ERROR TcpInputFd - SSL Error for fd from HOST:localhost.localdomain, IP:127.0.0.1, PORT:53075
splunkd.log:03-01-2010 13:35:28.653 ERROR TcpInputFd - SSL Error = error:140760FC:SSL routines:SSL23_GET_CLIENT_HELLO:unknown protocol
splunkd.log:03-01-2010 13:35:28.653 ERROR TcpInputFd - ACCEPT_RESULT=-1 VERIFY_RESULT=0
splunkd.log:03-01-2010 13:35:28.653 ERROR TcpInputFd - SSL Error for fd from HOST:localhost.localdomain, IP:127.0.0.1, PORT:53076
splunkd.log:03-01-2010 13:35:28.653 ERROR TcpInputFd - SSL Error = error:140760FC:SSL routines:SSL23_GET_CLIENT_HELLO:unknown protocol
splunkd.log:03-01-2010 13:35:28.654 ERROR TcpInputFd - ACCEPT_RESULT=-1 VERIFY_RESULT=0
splunkd.log:03-01-2010 13:35:28.654 ERROR TcpInputFd - SSL Error for fd from HOST:localhost.localdomain, IP:127.0.0.1, PORT:53077
splunkd.log:03-01-2010 13:35:28.654 ERROR TcpInputFd - SSL Error = error:140760FC:SSL routines:SSL23_GET_CLIENT_HELLO:unknown protocol
splunkd.log:03-01-2010 13:35:28.654 ERROR TcpInputFd - ACCEPT_RESULT=-1 VERIFY_RESULT=0

Closed receiver socket

If a receiving indexer's queues become full, it will close the receiver socket, to prevent additional forwarders from connecting to it. If a forwarder with load-balancing enabled can no longer forward to that receiver, it will send its data to another indexer on its list. If the fowarder does not employ load-balancing, it will hold the data until the problem is resolved.

The receiver socket will reopen automatically when the queue gets unclogged.

Typically, a receiver gets behind on the dataflow because it can no longer write data due to a full disk or because it is itself attempting to forward data to another Splunk instance that is not accepting data.

The following warning message will appear in splunkd.log if the socket gets blocked:

Stopping all listening ports. Queues blocked for more than N seconds. 

This message will appear when the socket reopens:

Started listening on tcp ports. Queues unblocked. 

Answers

Have questions? Visit Splunk Answers and see what questions and answers the Splunk community has around configuring forwarding.

PREVIOUS
Set up forwarding and receiving
  NEXT
Configure forwarders with outputs.conf

This documentation applies to the following versions of Splunk® Enterprise: 4.3, 4.3.1, 4.3.2, 4.3.3, 4.3.4, 4.3.5, 4.3.6, 4.3.7, 5.0, 5.0.1, 5.0.2, 5.0.3, 5.0.4, 5.0.5, 5.0.6, 5.0.7, 5.0.8, 5.0.9, 5.0.10, 5.0.11, 5.0.12, 5.0.13, 5.0.14, 5.0.15, 5.0.16, 5.0.17, 5.0.18


Comments

It appears that Skurylo's problem was due to a previously disabled version of that stanza in another copy of the inputs.conf file, which was overriding his more recent setting. <br /><br />For safety's sake, it doesn't hurt to add "disabled = 0" to the stanza (as now indicated in the text of this topic), to ensure that your configuration takes effect and is not blocked by a previous disablement. <br /><br />But, for the record, "[splunktcp://:9997]" and "[splunktcp://9997]" are syntactically identical in the Splunk conf system. You can use either form.

Sgoodman, Splunker
August 29, 2013

Sgoodman - I'm working on a new install and if I only put in a single colon, it doesn't work. I have to do [splunktcp://:9997] and after a restart netstat shows splunk listening.<br /><br />I'm running:<br /># /opt/splunk/bin/splunk version<br />Splunk 5.0.4 (build 172409)

Skurylo
August 28, 2013

Yoho: Actually, it's correct without the prepended ':'. That is, '[splunktcp://9997]'. The inputs.conf.spec file incorrectly states the syntax; I will make sure that gets changed.

Sgoodman, Splunker
July 25, 2013

I may be wrong but I have the impression the stanza in "Set up receiving with the configuration file" is wrong : shouldn't it be "[splunktcp://:9997]" instead of "[splunktcp://9997]" (additional ":") as the port number should be prepended with ":" ?

Yoho
May 27, 2013

Eegilbert - There are a few ways to specify which index the data goes to, depending on exactly what you want to accomplish. For complete details, see this subtopic: http://docs.splunk.com/Documentation/Splunk/latest/Indexer/Setupmultipleindexes#Send_events_to_specific_indexes<br /><br />If you want all of an input's events to go to a specific, non-default index, specify that index in the inputs.conf file on the forwarder(s) where you define the input. (You also need to create the index itself via a stanza in indexes.conf on the receiver, but don't fiddle with inputs.conf on the receiver.) <br /><br />But if instead you want to route some data to one index and some to another, you configure the routing by editing transforms.conf and props.conf on the receiver.

Sgoodman, Splunker
December 13, 2012

So, how does the receiver input relate to the data input? If one configures a receiver, how does one then specify which index to place this data into?

Eegilbert
December 13, 2012

Was this documentation topic helpful?

Enter your email address, and someone from the documentation team will respond to you:

Please provide your comments here. Ask a question or make a suggestion.

You must be logged into splunk.com in order to post comments. Log in now.

Please try to keep this discussion focused on the content covered in this documentation topic. If you have a more general question about Splunk functionality or are experiencing a difficulty with Splunk, consider posting a question to Splunkbase Answers.

0 out of 1000 Characters