Troubleshoot the universal forwarder

See common Splunk Universal Forwarder errors and how to fix them. For more troubleshooting information, check out the Splunk Community.

Warning appears in the universal forwarder when you run an SPL command

When you run an SPL command in the universal forwarder, the following messages may appear:

• Warning: Attempting to revert the SPLUNK_HOME ownership
• Warning: Executing "chown -R splunk /opt/splunkforwarder".

These warning do not affect functionality and can be ignored.

Splunk isn't receiving data from the universal forwarder

1. In the indexer user interface, go to forwarding and receiving, or go to inputs.conf.
2. Identify or select a port in Received Data to listen to. Make sure it is the same port set in outputs.conf for the forwarder to send data to. See Configure the universal forwarder using configuration files. Usually, the port 9997 splunktcp is preferred.
3. Check that the destination host for your indexers, including the IP address and hostname, is correct in outputs.conf.
4. After configuring your change, restart your Universal Forwarder. See Start or stop the Universal Forwarder.

Splunk is only receiving "\x00\" data

1. Go to your indexer user interface.
2. Ensure you are receiving data from Forwarding and receiving in indexer settings, and not Data inputs -> TCP/UDP.

Ingestion lagging

The most common cause of ingestion lagging is that you are taking in too much data from one sourcetype, which is blocking data from other sourcetypes. You can solve this by shortening your data ingestion intervals using the universal forwarder user interface, or inputs.conf.

Problems running 9.1 with older versions of indexers

Version 3 of the Splunk-to-Splunk protocol is deprecated as of version 9.0.0. If you use version 3 of the Splunk-to-Splunk protocol by setting negotiateProtocolLevel=0, then by default the forwarder switches to the latest Splunk-to-Splunk protocol in order to connect with other Splunk platform instances. The forwarder will also then generate warning logs.

Here are some example warning logs:

10-05-2022 21:14:48.078 +0000 WARN AutoLoadBalancedConnectionStrategy
[10422 TcpOutEloop] - Forwarder configured to use protocol level=0,
which is no longer supported, will use the lowest supported protocol level=1
10-05-2022 21:14:48.078 +0000 WARN AutoLoadBalancedConnectionStrategy
[10422 TcpOutEloop] - Indexer configured to use protocol level=0,
which is no longer supported, will use the lowest supported protocol level=1

To enable version 3 of Splunk-to-Splunk protocol, add enableOldS2SProtocol = true into the outputs.conf in the top [tcpout] stanza:

[tcpout]
enableOldS2SProtocol = true