Troubleshoot the Unix and Linux entity integration in ITSI

Here are some common *nix integration issues and how to resolve them.

collectd isn't sending metrics data to Splunk

Follow these steps to debug any collectd related issues:

1. Make sure a supported version of collectd is installed. To find the supported versions, see collectd support for *nix hosts.
2. If the version is correct, make sure the collectd process is running.
3. Once collectd is running or if it quits after it's started, check the collectd logs at /etc/collectd/collectd.log.
4. If there's a configuration file error, try to fix the collectd.conf file. For more information, see collectd package sources, install commands, and locations for ITSI. Try to disable the collectd plugin that has issues by commenting out the Loadplugin <plugin> stanza, then restart collectd.
5. Check the write_splunk configuration in collectd.conf located at /etc/collectd/collectd.conf or /etc/collectd.conf. Make sure all the configurations like token, port, and so on are correct.
6. Try sending fake data from the monitored *nix machine running collectd using curl –k>. For more information, see Example of sending metrics using HEC in the Splunk Enterprise Metrics manual. If this doesn't work, try to fix the network issue using the error message.
7. Check the HEC input at Settings > Data Inputs > HTTP Event Collector.
   • Verify the HEC token being used has the default index itsi_im_metrics.
   • Check the Global Settings for HEC. Verify that Enable SSL is checked and Use Deployment Server is unchecked. Also verify that the HEC port is the same as the one in collectd.conf. The port is generally 443 for Cloud HEC.

Splunk Add-on for Unix and Linux isn't sending metrics data to Splunk

1. Make sure the required dependencies for the add-on are installed. For more information, see Hardware and software requirements for the Splunk Add-on for Unix and Linux.
2. Check the inputs.conf file at $SPLUNK_HOME/etc/apps/Splunk_TA_nix/local/ and verify that metrics inputs are enabled and sending data to the correct metrics index. For a list of supported metrics inputs, see Enable data and scripted inputs for the Splunk Add-on for Unix and Linux.
3. Make sure the outputs.conf file on the universal forwarder is configured correctly. You can check your universal forwarder configuration in Splunk Web under Settings > Forwarding and receiving. Depending on your configuration you can also check the following locations for your universal forwarder configuration: $SPLUNK_HOME/etc/apps/SplunkUniversalForwarder/local, $SPLUNK_HOME/etc/apps/Splunk_TA_nix/local/, and $SPLUNK_HOME/etc/system/local/.
4. Make sure you're using the correct version of the Splunk Add-on for Unix and Linux and the universal forwarder. Metrics support was added to the add-on starting with version 8.1.0.
5. For additional troubleshooting, see Troubleshoot the Splunk Add-on for Unix and Linux.

collectd - Metrics data is in the index but there are no entities in ITSI

1. Make sure CPU metrics are available for the monitored host. collectd entity discovery uses the prefix cpu.* for metric names. Use mstats to look into the metrics data.
2. Make sure there's no data lag while indexing. If there's significant data lag, increase the dispatch.earliest_time setting and both earliest values in the search parameter to match in the [ITSI Import Objects - OS] stanza in $SPLUNK_HOME/etc/apps/itsi/local/savedsearches.conf.
3. Make sure data is indexed in the itsi_im_metrics index. If you're using a custom index, make sure the itsi_im_metrics_indexes macro is updated to include the custom index. For more information, see Use custom indexes in ITSI.
4. Make sure the entity discovery saved searches are enabled for the [ITSI Import Objects - OS] stanza in $SPLUNK_HOME/etc/apps/itsi/local/savedsearches.conf.

*nix Add-on - Metrics data is in the index but there are no entities in ITSI

1. Make sure cpu_metric metrics are available for the monitored host. Entity discovery in the Splunk Add-on for Unix and Linux uses the prefix cpu_metric.* for metric names. Use mstats to look into the metrics data.
2. Make sure there's no data lag while indexing. If there's significant data lag, increase the monitoring_window for the [ITSI Import Objects - TA *Nix] stanza in $SPLUNK_HOME/etc/apps/itsi/local/savedsearches.conf, then restart Splunk.
3. Make sure data is indexed in the itsi_im_metrics index. If you're using a custom index, make sure the itsi_im_metrics_indexes search macro is updated to include the custom index used. For more information, see Use custom metric indexes in ITSI.
4. Make sure the entity discovery saved searches are enabled for the [ITSI Import Objects - TA *Nix] stanza in $SPLUNK_HOME/etc/apps/itsi/local/savedsearches.conf.

*nix Add-on - Higher skipped search count than expected for ITSI Import Objects

Modify the SPL (Splunk Search Processing Language) to replace the out-of-the-box search stanza with the following saved search, using a dimension value specific to your environment:

| mcatalog values("host") as "dimension.identifier.host" 
 [ mcatalog values(_dims) as info where metric_name=*_metric.* AND (`itsi_entity_type_ta_nix_metrics_indexes`) earliest=-90s 
 | append [ | makeresults | eval info="no-data-placeholder"| fields - _time]| mvexpand info 
 | search info="IP_address" OR info="entity_type" OR info="IPv6_address" OR info="OS_name" OR info="location" OR info="os" OR info="ip"OR info="tag" OR info ="server"| 
 eval search="values(" . "\"" . info . "\"" . ") as " . "\"" . "dimension.info." . info . "\""| fields search| 
 mvcombine search| nomv search] where metric_name=*_metric.* AND (`itsi_entity_type_ta_nix_metrics_indexes`) earliest=-90s by "host" 
 | fields dimension.* 
 | eval identifier_dimensions="host" 
 | foreach dimension.*[| eval is_identifier=if(match("<<MATCHSTR>>", "identifier"), 1, 0) 
 | eval dimension_key=substr("<<MATCHSTR>>", len(if(is_identifier=1, "identifier.", "info.")) + 1)] 
 | rename dimension.identifier.* AS *, dimension.info.* AS * 
 | eval itsi_entity_id=host, etype="Unix/Linux Add-on", ip=IP_address

This saved search is more than twice as fast as the search provided out-of-the-box.