Troubleshoot the universal forwarder

See common Splunk Universal Forwarder errors and how to fix them. For more troubleshooting information, see https://community.splunk.com/t5/Community/ct-p/en-us/.

Warning appears in the universal forwarder when you run an SPL command

When you run an SPL command in the universal forwarder, the following messages may appear:

• Warning: Attempting to revert the SPLUNK_HOME ownership
• Warning: Executing "chown -R splunk /opt/splunkforwarder".

These warning do not affect functionality and can be ignored.

Splunk isn't receiving data from the universal forwarder

1. In the indexer user interface, go to forwarding and receiving, or go to inputs.conf.
2. Identify or select a port in Received Data to listen to. Make sure it is the same port set in outputs.conf for the forwarder to send data to. See Configure the universal forwarder using configuration files. Usually, the port 9997 splunktcp is preferred.
3. Check that the destination host for your indexers, including the IP address and hostname, is correct in outputs.conf.
4. After configuring your change, restart your Universal Forwarder. See Start or stop the Universal Forwarder.

Splunk is only receiving "\x00\" data

1. Go to your indexer user interface.
2. Ensure you are receiving data from Forwarding and receiving in indexer settings, and not Data inputs -> TCP/UDP.

Ingestion lagging

The most common cause of ingestion lagging is that you are taking in too much data from one sourcetype, which is blocking data from other sourcetypes. You can solve this by shortening your data ingestion intervals using the universal forwarder user interface, or inputs.conf.

Duplicate data or low disk space

When you set up multiple output groups in multiple stanzas using wildcards, the same data could be sent to all of the output groups. This could cause data to duplicate multiple times, which could increase the amount of disk space used and add additional work in the cluster.

If not monitored appropriately, the additional data could cause your hard disks to fill up and Splunk to stop working.

To mitigate this, reduce duplication so that all three of the following stanzas do not use wildcards:

In $SPLUNK_HOME/etc/apps/SplunkUniversalForwarder/default/inputs.conf:

[monitor://$SPLUNK_HOME\var\log\splunk\splunkd.log]
_TCP_ROUTING = *
index = _internal

[monitor://$SPLUNK_HOME\var\log\splunk\metrics.log]
_TCP_ROUTING = *
index = _internal

In $SPLUNK_HOME/etc/system/default/inputs.conf:

[monitor://$SPLUNK_HOME\etc\splunk.version]
_TCP_ROUTING = *
index = _internal
sourcetype=splunk_version