Splunk® Enterprise

Troubleshooting Manual

Download manual as PDF

This documentation does not apply to the most recent version of Splunk. Click here for the latest version.
Download topic as PDF

Contact Support

This page is currently a work in progress; expect frequent near-term updates.

For contact information, see the main Support contact page.

Here is some information on tools and techniques Splunk Support uses to diagnose problems. Many of these you can try yourself.

Note: Before you send any files or information to Splunk Support, verify that you are comfortable with sending it to us. We try to ensure that no sensitive information is included in any output from the commands below and in "Anonymize data samples to send to Support" in this manual, but we cannot guarantee compliance with your particular security policy.


The diag command collects basic info about your Splunk server, including Splunk's configuration details (such as the contents of $SPLUNK_HOME$/etc and general details about your index, like the host and source names). It does not include any event data or private information.

Be sure to run diag as a user with appropriate access to read Splunk files.


On *nix, in $SPLUNK_HOME$/bin

./splunk diag

On Windows in %SPLUNK_HOME%/bin

splunk diag

If you have difficultly running diag in your environment, you can also run the python script directly from the bin directory using cmd.

On *nix:

./splunk cmd python $SPLUNK_HOME/lib/python2.7/site-packages/splunk/clilib/info_gather.py

On Windows:

splunk cmd python %SPLUNK_HOME%\Python-2.7\Lib\site-packages\splunk\clilib\info_gather.py

This produces diag-<server name>-<date>.tar.gz, which you can send to Splunk Support for troubleshooting. If you're having trouble with forwarding, Support will probably need to see a diag for both your forwarder and your receiver.

Note: Before you upload, make sure the user who uploads the file has read permissions to the diag*.tar.gz file.

Upload your diag output to your Support case here:

Exclude files from a diag

Diag can be told to leave some files out of the diag with the switch --exclude. For example:

splunk diag --exclude "*/passwd"

This is repeatable:

splunk diag --exclude "*/passwd" --exclude "*/dispatch/*"

Defaults can also be controlled in server.conf. Refer to server.conf.spec in the Admin Manual for more information.

Run diag on a remote node

If you are not able to SSH into every machine in your deployment, you can still gather diags.

First, make sure you have the "get-diag" capability. Admin users have this capability. If admin users want to delegate this responsibility, they can give power users the get-diag capability.

You also need login credentials for the remote server.

The syntax is:

splunk diag -uri https://<host>:<mgmtPort>

Core Files

To collect a core file if Support asks you for one, use ulimit to remove any maximum file size setting before starting Splunk.

# ulimit -c unlimited

# splunk restart

This setting only affects the processes you start in a particular shell, so you may wish to do it in a new session. For Linux, start Splunk with the --nodaemon option (splunk start --nodaemon). In another shell, start the web interface manually with splunk start splunkweb.

Depending on your system, the core may be named something like core.1234, where the number indicates the process ID and is the same location as the splunkd executable.

LDAP configurations

If you are having trouble setting up LDAP, Support will typically need the following information:

  • The authentication.conf file from $SPLUNK_HOME/etc/system/local/.
  • An ldif for a group you are trying to map roles for.
  • An ldif for a user you are trying to authenticate as.

In some instances, a debug splunkd.log or web_service.log is helpful.

Upload your diag file for Support

Note: Before you upload, make sure the user who uploads the file has read permissions to the diag*.tar.gz file.

Upload your diag output to your Support case here:

Troubleshoot inputs with metrics.log
How to file a great Support case

This documentation applies to the following versions of Splunk® Enterprise: 6.0

Was this documentation topic helpful?

Enter your email address, and someone from the documentation team will respond to you:

Please provide your comments here. Ask a question or make a suggestion.

You must be logged into splunk.com in order to post comments. Log in now.

Please try to keep this discussion focused on the content covered in this documentation topic. If you have a more general question about Splunk functionality or are experiencing a difficulty with Splunk, consider posting a question to Splunkbase Answers.

0 out of 1000 Characters