This documentation does not apply to the most recent version of Splunk. Click here for the latest version.
- Diag options
- Change logging levels for individual subsystems from within Manager
- Enable debug logging for all logs by restarting Splunk in debug mode
- Enable debug messages in splunkd.log dynamically with a search (4.1.3 and earlier versions)
- Enable debug messages from the CLI (4.1.4 and later versions)
For contact information, see the main Support contact page.
Here is some information on tools and techniques Splunk Support uses to diagnose problems. Many of these you can try yourself.
Note: Before you send any files or information to Splunk Support, verify that you are comfortable with sending it to us. We try to ensure that no sensitive information is included in any output from the commands below, but we cannot guarantee compliance with your particular security policy.
The diag command collects basic info about your Splunk server, including Splunk's configuration details (such as the contents of
$SPLUNK_HOME/etc and general details about your index such as host and source names). It does not include any event data or private information.
Be sure to run diag as a user with appropriate access to read Splunk files.
If you have difficultly running diag in your environment, you can also run the python script directly from the bin directory using cmd
./splunk cmd python ../lib/python2.6/site-packages/splunk/clilib/info_gather.py
diag-<server name>-<date>.tar.gz (or .zip) that you can send to Splunk Support for troubleshooting.
Note: Before you upload, make sure the user who uploads the file has read permissions to the diag*.tar.gz file.
Upload your diag output to your Support case here -
- Enterprise Support: http://www.splunk.com/index.php/track_issues
- Community Members: http://www.splunk.com/index.php/send_to_splunk
- File exclusion
Diag can be told to leave some files out of the diag with the switch --exclude, for example:
splunk diag --exclude "*/etc/auth/splunk.secret"
Defaults can also be controlled in server.conf. Refer to server.conf.spec for more information.
Log levels and starting in debug mode
Splunk logging levels can be changed to provide more detail for different features either from within Splunk Manager, or by editing
Splunk's logging levels are
DEBUG INFO NOTICE WARN ERROR CRIT ALERT FATAL EMERG (most to least verbose). If a default level is not specified for a category the logging level defaults to your rootCategory setting. The easiest way to increase the verbosity of the log is to enable all messages with the --debug option. This does impact performance and should not be used routinely.
Change logging levels for individual subsystems from within Manager
- Log into Splunk Web as a user with admin role privileges.
- Click Manager > System settings > System logging.
- Select the log channel or channels you are interested in and adjust the logging level as desired.
- Click Save.
Enable debug logging for all logs by restarting Splunk in debug mode
- Stop Splunk, if it is running.
- Save your existing
splunkd.logfile by moving it to a new filename, like
- Restart Splunk in debug mode with
splunk start --debug.
- When you notice the problem, stop Splunk.
- Move the new
splunkd.logfile elsewhere and restore your old one.
- Restart Splunk normally (without the --debug flag) to disable debug logging.
Specific areas can be enabled to collect debugging details over a longer period with minimal performance impact. See the category settings in the file
$SPLUNK_HOME/etc/log.cfg to set specific log levels without enabling a large number of categories as with --debug. Restart Splunk after changing this file.
Important: Changes to
$SPLUNK_HOME/etc/log.cfg are overwritten on upgrade.
Note: Not all messages marked WARN or ERROR indicate actual problems with Splunk; some indicate that a feature is not being used.
Enable debug messages in splunkd.log dynamically with a search (4.1.3 and earlier versions)
To enable debugging, execute the following search
| debug cmd=logchange param1=root param2=DEBUG
To return to the default log level, execute the following search:
| debug cmd=logchange param1=root param2=WARN
To set a particular category of messages, replace "root" with the desired category. This does not change any settings in
log.cfg. On restart, the log level reverts to what is defined in
Note This search will return a "Error in 'DebugCommand': setting root priority" message. This is not an error and is normal. It is posted as an error message to ensure it is logged at any debug level.
Enable debug messages from the CLI (4.1.4 and later versions)
./splunk _internal call /server/logger/TailingProcessor -post:level DEBUG
Note This search will return a "HTTP Status: 200" message. This is not an error and is normal.
For investigating problems monitoring files, use the FileInputTracker and selectProcessor categories. These are not enabled with the normal "--debug" option because they are very verbose.
Debug Splunk Web
Change the logging level for Splunkweb by editing the file:
$SPLUNK_HOME/etc/log.cfgor if you have created your own
[python] stanza and change the contents to:
[python] splunk = DEBUG # other lines should be removed
The logging component names are hierarchical so setting the top level
splunk component will affect all loggers unless a more specific setting is provided, like
splunk.search = INFO
Restart the splunkweb process with the command ./splunk restart splunkweb. The additional messages are output in
To collect a core file, use ulimit to remove any maximum file size setting before starting Splunk.
# ulimit -c unlimited
# splunk restart
This setting only affects the processes you start in a particular shell, so you may wish to do it in a new session. For Linux, start Splunk with the --nodaemon option (
splunk start --nodaemon). In another shell, start the web interface manually with
splunk start splunkweb.
Depending on your system, the core may be named something like core.1234, where the number indicates the process id and be the same location as the splunkd executable.
If you are having trouble setting up LDAP, Support will typically need the following information:
- An ldif for a group you are trying to map roles for.
- An ldif for a user you are trying to authenticate as.
In some instances, a debug
web_service.log is helpful.
Recover metadata for a corrupt Splunk index directory
Important: You must contact Splunk support for direction before using this command.
recover-metadata command recovers missing or corrupt metadata associated with any Splunk index directory, sometimes also referred to as a 'bucket'. If your Splunk instance will not start up, one possible diagnosis is that one or more of your index buckets is corrupt in some way. Contact support; they will help you determine if this is indeed the case and if so, which bucket(s) are affected. Then, run this command:
$SPLUNK_HOME/bin/splunk cmd recover-metadata <full path to the exact index directory/bucket>
Splunk will return a success or failure message.