Troubleshooting Manual

 


Contact Support

NOTE - Splunk version 4.x reached its End of Life on October 1, 2013. Please see the migration information.

This documentation does not apply to the most recent version of Splunk. Click here for the latest version.

Contact Support

For contact information, see the main Support contact page.

Here is some information on tools and techniques Splunk Support uses to diagnose problems. Many of these you can try yourself.

Note: Before you send any files or information to Splunk Support, verify that you are comfortable with sending it to us. We try to ensure that no sensitive information is included in any output from the commands below and in "Anonymize data samples to send to Support" in this manual, but we cannot guarantee compliance with your particular security policy.

diag

The diag command collects basic info about your Splunk server, including Splunk's configuration details (such as the contents of $SPLUNK_HOME/etc and general details about your index, like the host and source names). It does not include any event data or private information.

Be sure to run diag as a user with appropriate access to read Splunk files.

Run:

On *nix: $SPLUNK_HOME/bin

./splunk diag

On Windows: %SPLUNK_HOME%/bin

splunk diag


If you have difficultly running diag in your environment, you can also run the python script directly from the bin directory using cmd.

On *nix:

./splunk cmd python $SPLUNK_HOME/lib/python2.7/site-packages/splunk/clilib/info_gather.py

On Windows:

splunk cmd python %SPLUNK_HOME%\Python-2.7\Lib\site-packages\splunk\clilib\info_gather.py

This produces diag-<server name>-<date>.tar.gz (or .zip), which you can send to Splunk Support for troubleshooting. If you're having trouble with forwarding, Support will probably need to see a diag for both your forwarder and your receiver.

Note: Before you upload, make sure the user who uploads the file has read permissions to the diag*.tar.gz file.

Upload your diag output to your Support case here:

Excluding files from a diag

Diag can be told to leave some files out of the diag with the switch --exclude. For example:

splunk diag --exclude "*/passwd"

This is repeatable:

splunk diag --exclude "*/passwd" --exclude "*/dispatch/*"

Defaults can also be controlled in server.conf. Refer to server.conf.spec in the Admin Manual for more information.

Core Files

To collect a core file if Support asks you for one, use ulimit to remove any maximum file size setting before starting Splunk.

# ulimit -c unlimited

# splunk restart

This setting only affects the processes you start in a particular shell, so you may wish to do it in a new session. For Linux, start Splunk with the --nodaemon option (splunk start --nodaemon). In another shell, start the web interface manually with splunk start splunkweb.

Depending on your system, the core may be named something like core.1234, where the number indicates the process ID and is the same location as the splunkd executable.

LDAP configurations

If you are having trouble setting up LDAP, Support will typically need the following information:

  • The authentication.conf file from $SPLUNK_HOME/etc/system/local/.
  • An ldif for a group you are trying to map roles for.
  • An ldif for a user you are trying to authenticate as.

In some instances, a debug splunkd.log or web_service.log is helpful.

Upload your diag file for Support

Note: Before you upload, make sure the user who uploads the file has read permissions to the diag*.tar.gz file.

Upload your diag output to your Support case here:

This documentation applies to the following versions of Splunk: 4.3 , 4.3.1 , 4.3.2 , 4.3.3 , 4.3.4 , 4.3.5 , 4.3.6 , 4.3.7 , 5.0 , 5.0.1 , 5.0.2 , 5.0.3 , 5.0.4 , 5.0.5 , 5.0.6 View the Article History for its revisions.


You must be logged into splunk.com in order to post comments. Log in now.

Was this documentation topic helpful?

If you'd like to hear back from us, please provide your email address:

We'd love to hear what you think about this topic or the documentation as a whole. Feedback you enter here will be delivered to the documentation team.

Feedback submitted, thanks!