Splunk® Enterprise

Forwarding Data

Acrobat logo Download manual as PDF


Splunk Enterprise version 8.0 is no longer supported as of October 22, 2021. See the Splunk Software Support Policy for details. For information about upgrading to a supported version, see How to upgrade Splunk Enterprise.
This documentation does not apply to the most recent version of Splunk® Enterprise. For documentation on the most recent version, go to the latest release.
Acrobat logo Download topic as PDF

Troubleshoot forwarder/receiver connection

If forwarding does not work, or does not work correctly, you can follow these troubleshooting steps to determine why.

Receiver does not accept new connections on its receiving port

If the internal queue on the receiving indexer gets blocked, the indexer shuts down the receiving/listening (splunktcp) port after a specified interval of being unable to insert data into the queue. Once the queue is again able to start accepting data, the indexer reopens the port.

However, sometimes (on Windows machines only) the indexer is unable to reopen the port once its queue is unblocked. To remediate, you must restart the indexer.

If you have this issue, you can set the receiver stopAcceptorAfterQBlock attribute in inputs.conf to a higher value, so that it does not close the port as quickly. This attribute determines the amount of time the indexer waits before closing the port. The default is 300 seconds (five minutes).

If you use load-balanced forwarders, they switch their data stream to another indexer in the load-balanced group based on their time-out interval, set in outputs.conf with the writeTimeout attribute. This results in automatic failover when the receiving indexers have blocked queues.

Confusing the receiving and management ports

As part of setting up a forwarder, specify the receiver's hostname/IP_address and port. The forwarder uses these to send data to the receiver. Be sure to specify the port that was designated as the receiving port at the time the receiver was configured. If you mistakenly specify the receiver's management port, the receiver will generate an error similar to this:

splunkd.log:03-01-2010 13:35:28.653 ERROR TcpInputFd - SSL Error = error:140760FC:SSL routines:SSL23_GET_CLIENT_HELLO:unknown protocol
splunkd.log:03-01-2010 13:35:28.653 ERROR TcpInputFd - ACCEPT_RESULT=-1 VERIFY_RESULT=0
splunkd.log:03-01-2010 13:35:28.653 ERROR TcpInputFd - SSL Error for fd from HOST:localhost.localdomain, IP:127.0.0.1, PORT:53075
splunkd.log:03-01-2010 13:35:28.653 ERROR TcpInputFd - SSL Error = error:140760FC:SSL routines:SSL23_GET_CLIENT_HELLO:unknown protocol
splunkd.log:03-01-2010 13:35:28.653 ERROR TcpInputFd - ACCEPT_RESULT=-1 VERIFY_RESULT=0
splunkd.log:03-01-2010 13:35:28.653 ERROR TcpInputFd - SSL Error for fd from HOST:localhost.localdomain, IP:127.0.0.1, PORT:53076
splunkd.log:03-01-2010 13:35:28.653 ERROR TcpInputFd - SSL Error = error:140760FC:SSL routines:SSL23_GET_CLIENT_HELLO:unknown protocol
splunkd.log:03-01-2010 13:35:28.654 ERROR TcpInputFd - ACCEPT_RESULT=-1 VERIFY_RESULT=0
splunkd.log:03-01-2010 13:35:28.654 ERROR TcpInputFd - SSL Error for fd from HOST:localhost.localdomain, IP:127.0.0.1, PORT:53077
splunkd.log:03-01-2010 13:35:28.654 ERROR TcpInputFd - SSL Error = error:140760FC:SSL routines:SSL23_GET_CLIENT_HELLO:unknown protocol
splunkd.log:03-01-2010 13:35:28.654 ERROR TcpInputFd - ACCEPT_RESULT=-1 VERIFY_RESULT=0

Closed receiver socket

If a receiving indexer queues become full, it closes the receiver socket to prevent additional forwarders from connecting to it. If a forwarder with load-balancing enabled can no longer forward to that receiver, it sends its data to another indexer on its list. If the forwarder does not employ load-balancing, it holds the data until you resolve the problem.

The receiver socket reopens automatically when the queue gets unclogged.

Typically, a receiver gets behind on the data flow because it can no longer write data due to a full disk or because it is itself attempting to forward data to another Splunk Enterprise instance that is not accepting data.

The following warning message will appear in splunkd.log if the socket gets blocked:

Stopping all listening ports. Queues blocked for more than N seconds. 

This message will appear when the socket reopens:

Started listening on tcp ports. Queues unblocked. 

Disable receiving

To disable receiving through the CLI, run the splunk disable listen command:

splunk disable listen -port <port> -auth <username>:<password>

You can also disable receiving by deleting the [splunktcp] stanza from inputs.conf.

Answers

Have questions? Visit Splunk Answers and see what questions and answers the Splunk community has around configuring forwarding.

Last modified on 08 January, 2021
PREVIOUS
Forward data to third-party systems
 

This documentation applies to the following versions of Splunk® Enterprise: 7.0.0, 7.0.1, 7.0.2, 7.0.3, 7.0.4, 7.0.5, 7.0.6, 7.0.7, 7.0.8, 7.0.9, 7.0.10, 7.0.11, 7.0.13, 7.1.0, 7.1.1, 7.1.2, 7.1.3, 7.1.4, 7.1.5, 7.1.6, 7.1.7, 7.1.8, 7.1.9, 7.1.10, 7.2.0, 7.2.1, 7.2.2, 7.2.3, 7.2.4, 7.2.5, 7.2.6, 7.2.7, 7.2.8, 7.2.9, 7.2.10, 7.3.0, 7.3.1, 7.3.2, 7.3.3, 7.3.4, 7.3.5, 7.3.6, 7.3.7, 7.3.8, 7.3.9, 8.0.0, 8.0.1, 8.0.2, 8.0.3, 8.0.4, 8.0.5, 8.0.6, 8.0.7, 8.0.8, 8.0.9, 8.0.10


Was this documentation topic helpful?


You must be logged into splunk.com in order to post comments. Log in now.

Please try to keep this discussion focused on the content covered in this documentation topic. If you have a more general question about Splunk functionality or are experiencing a difficulty with Splunk, consider posting a question to Splunkbase Answers.

0 out of 1000 Characters