Log data is not displaying alongside metric data

You are not seeing any log data with your metric data. Why is this happening?

1. What's going on

What's going on

Details

collectd is required for system metrics, and the Splunk forwarder is required for log collection and forwarding

While Splunk App for Infrastructure (SAI) uses collectd for system metrics, it requires the Splunk forwarder for log collection and forwarding. As with collectd, the forwarder is installed as part of the installation script. The default configuration sends log data over TCP to the receiving instance where SAI is installed. If you are having trouble getting data in, the following Investigation steps will help you identify the Possible root causes of this issue.

If you are deploying data collection on the SAI instance, you will have both a splunk directory and a splunkforwarder directory. Splunkforwarder is the directory that applies to the following sections. For all other systems, you will only have a splunkforwarder directory.

2. Investigation steps

Investigate the issue using these steps
1. From the terminal for the host in question, check the running status of splunk Ubuntu / Debian / Redhat: ps -aux | grep splunk As an alternative: $SPLUNK_HOME/bin/
2. Check the splunkd log file: $SPLUNK_HOME/var/log/splunk/splunkd.log

3. Possible causes

Possible cause	Reasons for the issue, or suggestions to resolve the issue
Splunk is not running	Try starting splunk manually. Enter $SPLUNK_HOME/bin/ ./splunk start
Hostnames for metrics and logs are not the same	This happens because the the FQDN lookup can return a different value for the log forwarder and the metrics agent. The app currently uses hostname as the correlation ID for metrics and logs. Use one of the following two options to resolve this issue.
	Option 1: Turn off FQDN lookup on collectd. Using this method for turning off FQDN lookup on the collectd agent, turn FQDN off and restart collectd. This typically resolves the issue.
	Option 2: Update splunk server settings with entity title. If you have metrics coming in and the entity has been discovered, the entity title is what the forwarder needs to use to assign the correct hostname. 1. Copy the entity title from the Splunk Insight for Infrastructure instance from the entity lister page. 2. Go to $SPLUNK_HOME/etc/sytem/local/ 3. Open server.conf and change the serverName option to the entity title. Save and close. 4. Open inputs.conf and change the host option to the entity title. Save and close. 5. Restart the forwarder $SPLUNK_HOME/bin/ and ./splunk restart If you have deployed the data collection tools on the same server where the SAI instance is running, $SPLUNK_HOME will be the location of the Insight instance. In the same root directory, there will be a splunkforwarder/directory, which is the location where changes need to be made.
The forwarder is blocked by your firewall	The Splunk forwarder sends log data to the Splunk Instance using port 9997 on the receiving Insight instance. Make sure your network allows for the forwarder to send to this location and port. If using a firewall, ensure the following ports are exposed via the firewall on the SAI server. Use TCP incoming/outgoing for all ports. 8088 port to receive metric data from the agent 9997 port to receive log data from the universal forwarder 8000 port to access the SAI user interface 8089 port to access the SAI REST API (advanced use cases only)