Troubleshoot the Splunk Add-on for VMware ESXi Logs

Not getting esxilogs

Problem

Not getting esxilogs while forwarding it to indexers which are in a cluster in the on-premise deployment. You might also see this error message splunkd.log on indexers:

ERROR AggregatorMiningProcessor - Uncaught Exception in Aggregator, skipping an event:
Can't open DateParser XML configuration file
"/opt/splunk/etc/apps/Splunk_TA_esxilogs/default/syslog_datetime.xml": No such file or
directory - data_source="/data/log_files/syslog/<hostname>.log", data_host="<hostname>",
data_sourcetype="vmw-syslog"

Cause

While esxilogs are directly forwarded to indexers (which are in the cluster), splunkd.log on indexers will show this error:

Splunk is not able to find a custom timestamp parser file (syslog_datetime.xml) which is used to extract dates and timestamps from events.

The following parameter is set for this in props.conf file present in the Splunk_TA_esxilogs package:

DATETIME_CONFIG = /etc/apps/Splunk_TA_esxilogs/default/syslog_datetime.xml

As indexers are in the cluster, Splunk_TA_esxilogs on indexers would be installed under slave-apps (/etc/slave-apps/) hence the path wouldn't exist. Note that as per the deployment guideline for the Splunk Add-on for VMware ESXi logs, if you are forwarding the ESXi logs in the cloud environment, you have to install the "Splunk_TA_esxilogs" package on the intermediate forwarder, and thus it wouldn't be needed on the indexer. Therefore, this issue can't occur in the cloud environment.

Solution

1. On cluster master, create a local directory in the $SPLUNK_HOME/etc/master-apps/Splunk_TA_esxilogs directory, if not present.
2. Create a props.conf file in the $SPLUNK_HOME/etc/master-apps/Splunk_TA_esxilogs/local directory (if not present) and add one of the following settings:
   • This is the setting for a Splunk Enterprise instance lower than 9.x:
   [vmw-syslog]
DATETIME_CONFIG = /etc/slave-apps/Splunk_TA_esxilogs/default/syslog_datetime.xml
   • This is the setting for a Splunk Enterprise instance 9.x or higher:
   [vmw-syslog]
DATETIME_CONFIG = /etc/peer-apps/Splunk_TA_esxilogs/default/syslog_datetime.xml
3. Push bundle on indexers