Splunk® IT Service Intelligence

Entity Integrations Manual

Acrobat logo Download manual as PDF


Acrobat logo Download topic as PDF

Troubleshoot the Unix and Linux entity integration in ITSI

Here are some common *nix integration issues and how to resolve them.

collectd isn't sending metrics data to Splunk

Follow these steps to debug any collectd related issues:

  1. Make sure a supported version of collectd is installed. To find the supported versions, see collectd support for *nix hosts.
  2. If the version is correct, make sure the collectd process is running.
  3. Once collectd is running or if it quits after it's started, check the collectd logs at /etc/collectd/collectd.log.
  4. If there's a configuration file error, try to fix the collectd.conf file. For more information, see collectd package sources, install commands, and locations for ITSI. Try to disable the collectd plugin that has issues by commenting out the Loadplugin <plugin> stanza, then restart collectd.
  5. Check the write_splunk configuration in collectd.conf located at /etc/collectd/collectd.conf or /etc/collectd.conf. Make sure all the configurations like token, port, and so on are correct.
  6. Try sending fake data from the monitored *nix machine running collectd using curl –k>. For more information, see Example of sending metrics using HEC in the Splunk Enterprise Metrics manual. If this doesn't work, try to fix the network issue using the error message.
  7. Check the HEC input at Settings > Data Inputs > HTTP Event Collector.
    • Verify the HEC token being used has the default index em_metrics.
    • Check the Global Settings for HEC. Verify that Enable SSL is checked and Use Deployment Server is unchecked. Also verify that the HEC port is the same as the one in collectd.conf. The port is generally 443 for Cloud HEC.

The Splunk Add-on for Unix and Linux isn't sending metrics data to Splunk

  1. Make sure the required dependencies for the add-on are installed. For more information, see Hardware and software requirements for the Splunk Add-on for Unix and Linux.
  2. Check the inputs.conf file at $SPLUNK_HOME/etc/apps/splunk_app_infrastructure/local and verify that metrics inputs are enabled and sending data to the correct metrics index. For a list of supported metrics inputs, see Enable data and scripted inputs for the Splunk Add-on for Unix and Linux.
  3. Make sure the outputs.conf file at $SPLUNK_HOME/etc/apps/splunk_app_infrastructure/local on the universal forwarder is configured correctly.
  4. Make sure you're using the correct version of the Splunk Add-on for Unix and Linux and the universal forwarder. Metrics support was added to the add-on starting with version 8.1.0.
  5. For additional troubleshooting, see Troubleshoot the Splunk Add-on for Unix and Linux.

collectd - Metrics data is in the index but there are no entities in ITSI

  1. Make sure CPU metrics are available for the monitored host. collectd entity discovery uses the prefix cpu.* for metric names. Use mstats to look into the metrics data.
  2. Make sure there's no data lag while indexing. If there's significant data lag, increase the monitoring_window for [os] stanza in entity_classes.conf within SAI, then restart Splunk.
  3. Make sure data is indexed in the em_metrics index. If you're using a custom index, make sure the sai_metrics_indexes macro is updated to include the custom index. For more information, see Use custom metric indexes in Splunk App for Infrastructure in the Administer Splunk App for Infrastructure manual.
  4. Make sure the entity discovery saved searches are enabled for the [Entity Class - os] stanza in $SPLUNK_HOME/etc/apps/splunk_app_infrastructure/local/savedsearches.conf.

*nix Add-on - Metrics data is in the index but there are no entities in ITSI

  1. Make sure cpu_metric metrics are available for the monitored host. Entity discovery in the Splunk Add-on for Unix and Linux uses the prefix cpu_metric.* for metric names. Use mstats to look into the metrics data.
  2. Make sure there's no data lag while indexing. If there's significant data lag, increase the monitoring_window for the [ta_nix] stanza in $SPLUNK_HOME/etc/apps/splunk_app_infrastructure/local/entity_classes.conf, then restart Splunk.
  3. Make sure data is indexed in the em_metrics index. If you're using a custom index, make sure the sai_metrics_indexes macro is updated to include the custom index used. For more information, see Use custom metric indexes in Splunk App for Infrastructure in the Administer Splunk App for Infrastructure manual.
  4. Make sure the entity discovery saved searches are enabled for the [Entity Class - ta_nix] stanza in $SPLUNK_HOME/etc/apps/splunk_app_infrastructure/local/savedsearches.conf.

Entities appear in SAI but not in ITSI

Make sure the SAI entity integration in enabled. Entity discovery occurs in SAI and entities are sent to ITSI through the message bus. For more information, see Integrate the Splunk App for Infrastructure with ITSI.

Last modified on 14 December, 2020
PREVIOUS
Stop collecting data from a *nix host in ITSI
  NEXT
About the Windows entity integration in ITSI

This documentation applies to the following versions of Splunk® IT Service Intelligence: 4.6.0 Cloud only, 4.6.1 Cloud only, 4.6.2 Cloud only, 4.7.0, 4.7.1, 4.7.2, 4.8.0 Cloud only, 4.8.1 Cloud only, 4.9.0, 4.9.1, 4.9.2, 4.9.3, 4.10.0 Cloud only, 4.10.1 Cloud only, 4.10.2 Cloud only


Was this documentation topic helpful?

You must be logged into splunk.com in order to post comments. Log in now.

Please try to keep this discussion focused on the content covered in this documentation topic. If you have a more general question about Splunk functionality or are experiencing a difficulty with Splunk, consider posting a question to Splunkbase Answers.

0 out of 1000 Characters