Splunk® App for Infrastructure (Legacy)

Administer Splunk App for Infrastructure

On August 22, 2022, the Splunk App Infrastructure will reach its end of life and Splunk will no longer maintain or develop this product.

Log data is not displaying alongside metric data

You are not seeing any log data with your metric data. Why is this happening?

1. What's going on

What's going on Details
collectd is required for system metrics, and the Splunk forwarder is required for log collection and forwarding While Splunk App for Infrastructure (SAI) uses collectd for system metrics, it requires the Splunk forwarder for log collection and forwarding. As with collectd, the forwarder is installed as part of the installation script. The default configuration sends log data over TCP to the receiving instance where SAI is installed. If you are having trouble getting data in, the following Investigation steps will help you identify the Possible root causes of this issue.

If you are deploying data collection on the SAI instance, you will have both a splunk directory and a splunkforwarder directory. Splunkforwarder is the directory that applies to the following sections. For all other systems, you will only have a splunkforwarder directory.

2. Investigation steps

Investigate the issue using these steps
1. From the terminal for the host in question, check the running status of splunk
  • Ubuntu / Debian / Redhat: ps -aux | grep splunk
  • As an alternative: $SPLUNK_HOME/bin/
2. Check the splunkd log file:
  • $SPLUNK_HOME/var/log/splunk/splunkd.log

3. Possible causes

Possible cause Reasons for the issue, or suggestions to resolve the issue
Splunk is not running Try starting splunk manually. Enter $SPLUNK_HOME/bin/ ./splunk start
Hostnames for metrics and logs are not the same This happens because the the FQDN lookup can return a different value for the log forwarder and the metrics agent. The app currently uses hostname as the correlation ID for metrics and logs. Use one of the following two options to resolve this issue.
Option 1: Turn off FQDN lookup on collectd. Using this method for turning off FQDN lookup on the collectd agent, turn FQDN off and restart collectd. This typically resolves the issue.
Option 2: Update splunk server settings with entity title. If you have metrics coming in and the entity has been discovered, the entity title is what the forwarder needs to use to assign the correct hostname.

1. Copy the entity title from the Splunk Insight for Infrastructure instance from the entity lister page.

2. Go to $SPLUNK_HOME/etc/sytem/local/

3. Open server.conf and change the serverName option to the entity title. Save and close.

4. Open inputs.conf and change the host option to the entity title. Save and close.

5. Restart the forwarder $SPLUNK_HOME/bin/ and ./splunk restart

If you have deployed the data collection tools on the same server where the SAI instance is running, $SPLUNK_HOME will be the location of the Insight instance. In the same root directory, there will be a splunkforwarder/directory, which is the location where changes need to be made.

The forwarder is blocked by your firewall The Splunk forwarder sends log data to the Splunk Instance using port 9997 on the receiving Insight instance. Make sure your network allows for the forwarder to send to this location and port.

If using a firewall, ensure the following ports are exposed via the firewall on the SAI server. Use TCP incoming/outgoing for all ports.

  • 8088 port to receive metric data from the agent
  • 9997 port to receive log data from the universal forwarder
  • 8000 port to access the SAI user interface
  • 8089 port to access the SAI REST API (advanced use cases only)
Last modified on 08 July, 2020
Data collection is not working and entities are not displaying   Collectd DF Plugin not generating output on Linux with XFS file system

This documentation applies to the following versions of Splunk® App for Infrastructure (Legacy): 1.3.0, 1.3.1, 1.4.0, 1.4.1, 2.0.0, 2.0.1, 2.0.2, 2.0.3, 2.0.4, 2.1.0, 2.1.1 Cloud only, 2.2.0 Cloud only, 2.2.1, 2.2.3 Cloud only, 2.2.4, 2.2.5

Was this topic useful?

You must be logged into splunk.com in order to post comments. Log in now.

Please try to keep this discussion focused on the content covered in this documentation topic. If you have a more general question about Splunk functionality or are experiencing a difficulty with Splunk, consider posting a question to Splunkbase Answers.

0 out of 1000 Characters