Splunk® App for Infrastructure

Administer Splunk App for Infrastructure

Download manual as PDF

Download topic as PDF

Data collection is not working and entities are not displaying

The entities you have added are not displaying in the user interface, and it seems no data is being collected (Unix Data Collection). Why is this happening?

In some instances, it can take up to about five (5) minutes for initial entity discovery. Ensure that you have waited at least this amount of time before moving on to the next steps. If this is not the case, see the following information about what might be causing this issue.

1. What's going on

What's going on Details
collectd is not running, or has failed with errors Splunk App for Infrastructure (SAI) uses collectd to provide data collection and sending many common system performance metrics. The easy install script does most of the work for setting up and sending data, but not all systems are alike. If you are having trouble getting data in, the following Investigation steps will help you identify the Possible root causes of this issue.

2. Investigation steps

Investigate the issue using these steps
1. Use your terminal to ssh into the server in question.
2. Run a status check to ensure collectd is running.
3. Check the collectd status.
  • Debian/Ubuntu/Redhat/Centos: service collectd status
4. If there are no immediate status errors, check the log files:
  • Debian/Ubuntu: /etc/collectd/collectd.log
  • Redhat/Centos: /etc/collectd.log

3. Possible causes

Possible cause Reasons for the issue, or suggestions to resolve the issue
Missing or wrong dependencies Missing libcurl dependency. Installation script failed to install the libcurl dependency.
User does not have root privileges The installation was not run by a user with root privileges.
Unable to resolve hostname You have set the FQDNLookup option, but you cannot resolve your hostname to a fully qualified domain name. You need to fix the network configuration, as follows:

1. Go to collectd.conf, which is in the same directory as your collectd.log file.

2. Open the file with a text editor.

3. Uncomment FQDNLookup true

4. Change to FQDNLookup false

5. Restart collectd

  • Debian/Ubuntu/Redhat/Centos: service collectd restart
Agent data is blocked by a firewall The collectd daemon transmits metric data over HTTP. Your network must allow each host to send data to the receiving instance (where you installed SAI) on port 8088.

If using a firewall, ensure the following ports are exposed via the firewall on the SAI server. Use TCP incoming/outgoing for all ports.

  • 8088 port to receive metric data from the agent
  • 9997 port to receive log data from the universal forwarder
  • 8000 port to access the SAI user interface
  • 8089 port to access the SAI REST API (advanced use cases only)
Sending data to HEC over HTTP with SSL enabled If you're sending data to HEC over HTTP with SSL enabled, you'll receive a status 35: SSL connect error. Disable SSL if you're sending data to HEC over HTTP. Disable SSL in your collectd.conf file.
PREVIOUS
The easy install script repeatedly requests user credentials
  NEXT
Log data is not displaying alongside metric data

This documentation applies to the following versions of Splunk® App for Infrastructure: 1.3.0, 1.3.1, 1.4.0, 1.4.1


Comments

Hello! Thank you for the feedback. We added the SSL connect error and solution to this topic.

Bashby splunk, Splunker
October 8, 2019

Hi docs team - Troubleshooting my SAI collectd feed, I found the issue was HEC cert related (found the event in /etc/collectd/collectd.log)
Can we include a connectivity troubleshooting step to look for:
[2019-05-10 10:55:26] [error] write splunk plugin: curl_easy_perform failed to connect to my.hec.ip.addr:8088 with status 35: SSL connect error
[2019-05-10 10:55:26] [error] write_splunk plugin: post data failed

These logs aren't in Splunk as the endpoint couldn't reach there. I didn't know where to look on the endpoint having never touched collectd before and this hint would have saved me tons of time. Thanks!

Bcusick splunk, Splunker
October 7, 2019

Was this documentation topic helpful?

Enter your email address, and someone from the documentation team will respond to you:

Please provide your comments here. Ask a question or make a suggestion.

You must be logged into splunk.com in order to post comments. Log in now.

Please try to keep this discussion focused on the content covered in this documentation topic. If you have a more general question about Splunk functionality or are experiencing a difficulty with Splunk, consider posting a question to Splunkbase Answers.

0 out of 1000 Characters