Troubleshoot the Splunk Add-on for EMC VNX

General troubleshooting

For troubleshooting tips that you can apply to all add-ons, see Troubleshoot add-ons in Splunk Add-ons. For additional resources, see Support and resource links for add-ons in Splunk Add-ons.

Log files

The three log files that you can use to diagnose problems with this add-on are:

• ta_vnx.log
• data_loader.log
• splunkd.log

You can find these under $SPLUNK_HOME/var/log/splunk. Before the add-on successfully runs for the first time, error logs go to splunkd.log. After the add-on successfully runs, error logs go to ta_vnx.log and data_loader.log.

Set the loglevel to DEBUG in default/inputs.conf and restart splunkd to have more verbose diagnositic messages.

General debugging in *NIX systems

If you are on an *NIX system, you can go to $SPLUNK_HOME/etc/apps/Splunk_TA_emc-vnx/bin and run the following commands to check if any configuration errors are reported.

source $SPLUNK_HOME/bin/setSplunkEnv
which naviseccli

A successful result looks like this:

-bash-4.1$ source $SPLUNK_HOME/bin/setSplunkEnv && which naviseccli
Tab-completion of "splunk <verb> <object>" is available.
~/bin/naviseccli

A failed result looks like this:

-bash-4.1$ source $SPLUNK_HOME/bin/setSplunkEnv && which naviseccli
Tab-completion of "splunk <verb> <object>" is available.
/usr/bin/which: no naviseccli in (/opt/splunk/bin:/usr/lib64/qt-3.3/bin:/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin:/home/user/bin)

You can list inputs.conf stanzas with this command:

source $SPLUNK_HOME/bin/setSplunkEnv
$SPLUNK_HOME/bin/splunk cmd splunkd print-modinput-config vnx_data_loader

A successful result lists the configuration, as shown.

-bash-4.1$ source $SPLUNK_HOME/bin/setSplunkEnv && $SPLUNK_HOME/bin/splunk cmd splunkd print-modinput-config vnx_data_loader
<?xml version="1.0" encoding="UTF-8"?>
<input>
  <server_host>apps-demo02.sv.splunk.com</server_host>
  <server_uri>https://127.0.0.1:8089</server_uri>
  <session_key>JhC4AzvSLmJWJKG^1LSn^biPb^bj5CJx22hVYNyObJ7aNouCIjvpqRpQfNmtEo2jhF2F0qpys7RWaqK7dLRm8MA4YdBfE21t1XkeIqNuDbiX5Vl8ZtWg</session_key>
  <checkpoint_dir>/usr/local/bamboo/splunk/var/lib/splunk/modinputs/vnx_data_loader</checkpoint_dir>
  <configuration>
    <stanza name="vnx_data_loader://MYVNX_block">
      <param name="disabled">false</param>
      <param name="host">apps-demo02.sv.splunk.com</param>
      <param name="index">vnx_logs</param>
      <param name="interval">60</param>
      <param name="loglevel">DEBUG</param>
      <param name="network_addr">10.1.1.2</param>
      <param name="password"><encrypted></param>
      <param name="platform">VNX Block</param>
      <param name="site">SFCA</param>
      <param name="username"><encrypted></param>
    </stanza>
    <stanza name="vnx_data_loader://MYVNX_file">
      <param name="disabled">false</param>
      <param name="host">apps-demo02.sv.splunk.com</param>
      <param name="index">vnx_logs</param>
      <param name="interval">60</param>
      <param name="loglevel">WARN</param>
      <param name="network_addr">10.1.1.4</param>
      <param name="password"><encrypted></param>
      <param name="platform">VNX File</param>
      <param name="site">SFCA</param>
      <param name="username"><encrypted></param>
    </stanza>
  </configuration>
</input>
-bash-4.1$ 

You can validate inputs.conf stanzas with this command:

source $SPLUNK_HOME/bin/setSplunkEnv
$SPLUNK_HOME/bin/splunk cmd splunkd print-modinput-config vnx_data_loader | python vnx_data_loader.py --validate-arguments

A successful configuration validates and exit quietly, as shown.

-bash-4.1$ source $SPLUNK_HOME/bin/setSplunkEnv && $SPLUNK_HOME/bin/splunk cmd splunkd print-modinput-config vnx_data_loader | python vnx_data_loader.py --validate-arguments
-bash-4.1$ 

A misconfiguration prints errors, as shown.

-bash-4.1$ source $SPLUNK_HOME/bin/setSplunkEnv && $SPLUNK_HOME/bin/splunk cmd splunkd print-modinput-config vnx_data_loader | python vnx_data_loader.py --validate-arguments
-bash-4.1$ 
Traceback (most recent call last):
  File "vnx_data_loader.py", line 467, in <module>
    main()
  File "vnx_data_loader.py", line 454, in main
    validate_config()
  File "vnx_data_loader.py", line 388, in validate_config
    raise Exception("Missing network address in stanza: %s" % stanza)
Exception: Missing network address in stanza: vnx_data_loader://MYVNX_block

Control Stations still ask for passwords after pushing the public keys

If the Control Stations still ask for passwords after pushing the public keys, run the ssh in debug mode to check the possible issue.

ssh -v username@cs_host

If "Agent admitted failure to sign using the key" shows up in the debug output, run the following command on the collecting host to fix this.

ssh-add ~/.ssh/id_rsa

Data gaps

Splunk Enterprise should index the metric data per the interval set in ta_vnx_collection.conf. If you find a data gap, check work_queue_size in $SPLUNK_HOME/var/log/splunk/data_loader.log. If this value stays consistently over 10, it means the add-on is under load and thus failed to handle the data gathering job as fast as necessary. For example, if the VNX File performance data gathering interval is set to two minutes, but the add-on takes three minutes to finish the data gathering, there will be data gap as the program runs. The performance problems usually lie in the VNX Array/VNX File itself. The file and the array takes too much time to return the response to the add-on. If it is the latter case (which you can check by verifying the log in ta_vnx.log), increase the data collection interval by configuring ta_vnx_collection.conf.

Missing VNX File metrics in Windows

For Windows, if the VNX File metrics (sourcetype=vnx_file*) are missing for some VNX servers, check ta_vnx.log and look for errors with "Server refused our key". If there are, re-input the username and password in local\inputs.conf for the specific VNX files and restart splunkd.

Missing metrics from local network

If there are metrics missing, check ta_vnx.log to see if there are "timedout" errors. If there are, and the VNXes are in local network, the naviseccli/plink/ssh configuration for these VNXes may be flawed. Revisit the naviseccli/plink/ssh installation steps.