Troubleshoot Data Collection for Splunk App for Infrastructure

Use the following troubleshooting topics to help resolve data collection issues you might be having in your instance of Splunk App for Infrastructure:

• Data collection is not working and entities are not displaying
• Log data is not displaying alongside metric data

Data collection is not working and entities are not displaying

The entities you have added are not displaying in the user interface, and it seems no data is being collected (Unix Data Collection). Why is this happening?

In some instances, it can take up to about five (5) minutes for initial entity discovery. Ensure that you have waited at least this amount of time before moving on to the next steps. If this is not the case, see the following information about what might be causing this issue.

1. What's going on

What's going on

Details

collectd is not running, or has failed with errors

Splunk App for Infrastructure uses collectd to provide data collection and sending many common system performance metrics. The Splunk App for Infrastructure installation script does most of the work for setting up and sending data, but not all systems are alike. If you are having trouble getting data in, the following Investigation steps will help you identify the Possible root causes of this issue.

2. Investigation steps

Investigate the issue using these steps
1. Use your terminal to ssh into the server in question.
2. Run a status check to ensure collectd is running.
3. Check the collectd status. Debian/Ubuntu/Redhat/Centos: service collectd status
4. If there are no immediate status errors, check the log files: Debian/Ubuntu: /etc/collectd/collectd.log Redhat/Centos: /etc/collectd.log

3. Possible causes

Possible cause	Reasons for the issue, or suggestions to resolve the issue
Missing or wrong dependencies	Missing libcurl dependency. Installation script failed to install the libcurl dependency.
User does not have root privileges	The installation was not run by a user with root privileges.
Unable to resolve hostname	You have set the FQDNLookup option, but you cannot resolve your hostname to a fully qualified domain name. You need to fix the network configuration, as follows: 1. Go to collectd.conf, which is in the same directory as your collectd.log file. 2. Open the file with a text editor. 3. Uncomment FQDNLookup true 4. Change to FQDNLookup false 5. Restart collectd Debian/Ubuntu/Redhat/Centos: service collectd restart
Agent data is blocked by a firewall	The collectd daemon transmits metric data over HTTP. Your network must allow each host to send data to the receiving instance (where Splunk App for Infrastructure is installed) on port 8088. If using a firewall, ensure the following ports are exposed via the firewall on the Splunk App for Infrastructure server. Use TCP incoming/outgoing for all ports. 8088 port to receive metric data from the agent 9997 port to receive log data from the universal forwarder 8000 port to access the Splunk App for Infrastructure user interface 8089 port to access Splunk App for Infrastructure REST API (advanced use cases only)

Log data is not displaying alongside metric data

You are not seeing any log data with your metric data. Why is this happening?

1. What's going on

What's going on

Details

collectd is required for system metrics, and the Splunk forwarder is required for log collection and forwarding

While Splunk App for Infrastructure uses collectd for system metrics, it requires the Splunk forwarder for log collection and forwarding. As with collectd, the forwarder is installed as part of the installation script. The default configuration sends log data over TCP to the receiving instance where Splunk App for Infrastructure is installed. If you are having trouble getting data in, the following Investigation steps will help you identify the Possible root causes of this issue.

If you are deploying data collection on the Splunk App for Infrastructure instance, you will have both a splunk directory and a splunkforwarder directory. Splunkforwarder is the directory that applies to the following sections. For all other systems, you will only have a splunkforwarder directory.

2. Investigation steps

Investigate the issue using these steps
1. From the terminal for the host in question, check the running status of splunk Ubuntu / Debian / Redhat: ps -aux | grep splunk As an alternative: $SPLUNK_HOME/bin/
2. Check the splunkd log file: $SPLUNK_HOME/var/log/splunk/splunkd.log

3. Possible causes

Possible cause	Reasons for the issue, or suggestions to resolve the issue
Splunk is not running	Try starting splunk manually. Enter $SPLUNK_HOME/bin/ ./splunk start
Hostnames for metrics and logs are not the same	This happens because the the FQDN lookup can return a different value for the log forwarder and the metrics agent. The app currently uses hostname as the correlation ID for metrics and logs. Use one of the following two options to resolve this issue.
	Option 1: Turn off FQDN lookup on collectd. Using this method for turning off FQDN lookup on the collectd agent, turn FQDN off and restart collectd. This typically resolves the issue.
	Option 2: Update splunk server settings with entity title. If you have metrics coming in and the entity has been discovered, the entity title is what the forwarder needs to use to assign the correct hostname. 1. Copy the entity title from the Splunk App for Infrastructure instance from the entity lister page. 2. Go to $SPLUNK_HOME/etc/sytem/local/ 3. Open server.conf and change the serverName option to the entity title. Save and close. 4. Open inputs.conf and change the host option to the entity title. Save and close. 5. Restart the forwarder $SPLUNK_HOME/bin/ and ./splunk restart If you have deployed the data collection tools on the same server where the Splunk App for Infrastructure instance is running, $SPLUNK_HOME will be the location of the App instance. In the same root directory, there will be a splunkforwarder/directory, which is the location where changes need to be made.
The forwarder is blocked by your firewall	The Splunk forwarder sends log data to the Splunk Instance using port 9997 on the receiving App instance. Make sure your network allows for the forwarder to send to this location and port. If using a firewall, ensure the following ports are exposed via the firewall on the Splunk App for Infrastructure server. Use TCP incoming/outgoing for all ports. 8088 port to receive metric data from the agent 9997 port to receive log data from the universal forwarder 8000 port to access the Splunk App for Infrastructure user interface 8089 port to access Splunk App for Infrastructure REST API (advanced use cases only)