Splunk® Enterprise

Troubleshooting Manual

Download manual as PDF

Splunk version 4.x reached its End of Life on October 1, 2013. Please see the migration information.
This documentation does not apply to the most recent version of Splunk. Click here for the latest version.
Download topic as PDF

Enable debug logging

Splunk's internal logging levels are DEBUG INFO WARN ERROR CRIT FATAL (from most to least verbose). This topic gives a few popular options for how you might want to put Splunk into debug mode.

Be warned, Splunk's debug mode is extremely verbose. All the extra chatter might obscure something that might have helped you diagnose your problem. And running Splunk in debug mode for any length of time will make your internal log files really pretty unwieldy. Running debug mode is not recommended on production systems.

Enable debug logging on all of splunkd.log

Splunk has a debugging parameter (--debug) that you can use when starting Splunk from the CLI. This command outputs logs to $SPLUNK_HOME/var/log/splunk/splunkd.log. To enable debug logging from the command line:

  • Navigate to $SPLUNK_HOME/bin.
  • Stop Splunk, if it is running.
  • Save your existing splunkd.log file by moving it to a new filename, like splunkd.log.old.
  • Restart Splunk in debug mode with splunk start --debug.
  • When you notice the problem, stop Splunk.
  • Move the new splunkd.log file elsewhere and restore your old one.
  • Stop or restart Splunk normally (without the --debug flag) to disable debug logging.

Specific areas can be enabled to collect debugging details over a longer period with minimal performance impact. See the category settings in the file $SPLUNK_HOME/etc/log.cfg to set specific log levels without enabling a large number of categories as with --debug. Restart Splunk after changing this file.

Important: Changes to $SPLUNK_HOME/etc/log.cfg are overwritten if you upgrade your version of Splunk.

Note: Not all messages marked WARN or ERROR indicate actual problems with Splunk; some indicate that a feature is not being used.

Enable debug logging for a specific processor within splunkd.log

In Splunk Web

You can enable these DEBUG settings via Splunk Web if you have admin privileges. Navigate to Manager » System settings » System logging. Search for the processor names using the text box. Click on the processor name to change the logging level to DEBUG. You do not need to restart Splunk. In fact, these changes will not persist if you restart the Splunk instance.

In log.cfg

If you want the processors to be in DEBUG on startup, or if you want to turn on debugging for a few processors or for a lightweight forwarder (with no Splunk Web), edit the $SPLUNK_HOME/etc/log.cfg file directly. Back up your log.cfg file before making any changes.

In $SPLUNK_HOME/etc/log.cfg, find the category.* entry that relates to the processor you are interested in, and change the INFO or WARN string to DEBUG. There will not always be an existing entry for the processor you are interested in, and it may take some digging through the logs or documentation to find the correct one.

For example, to see how often Splunk is checking on a particular file, put 'FileInputTracker' in DEBUG. Update the existing entry to read
category.FileInputTracker=DEBUG

Or for investigating problems monitoring files, use the FileInputTracker and selectProcessor categories.

Restart Splunk. Now every time Splunk checks the inputs file, it will be recorded in $SPLUNK_HOME/var/log/splunk/splunkd.log. Remember to change these settings back when you are finished investigating.

If a default level is not specified for a category, the logging level defaults to your rootCategory setting.

Note: leave category.loader at INFO. This is what gives us our build and system info.

To change the maximum size of a log file before it rolls, change the maxFileSize value (in bytes) for the desired file:

appender.A1=RollingFileAppender
appender.A1.fileName=${SPLUNK_HOME}/var/log/splunk/splunkd.log
appender.A1.maxFileSize=250000000
appender.A1.maxBackupIndex=5
appender.A1.layout=PatternLayout
appender.A1.layout.ConversionPattern=%d{%m-%d-%Y %H:%M:%S.%l} %-5p %c - %m%n

log-local.cfg

You can put log.cfg settings into a local file, log-local.cfg file, residing in the same directory as log.cfg. The settings in log-local.cfg take precedence. And unlike log.cfg, the log-local.cfg file doesn't get overwritten on upgrade.

With endpoints

In Splunk 4.1 and later, you can access a debugging endpoint that shows status information about monitored files:

https://your-splunk-server:8089/services/admin/inputstatus/TailingProcessor:FileStatus

Enable debug messages from the CLI (4.1.4 and later versions)

./splunk _internal call /server/logger/TailingProcessor -post:level DEBUG

Note: This search will return the message "HTTP Status: 200". This is not an error and is normal.


From 4.2, you can also set this way;

./splunk set log-level TailingProcessor -level DEBUG

Enable debug logging for search processes

Search processes obey the etc/log-searchprocess.cfg rules. Similar to splunkd, they can be overridden in etc/log-searchprocess-local.cfg.

Set all loggers to DEBUG by adding a line such as

rootCategory=DEBUG,searchprocessAppender

You can set specific loggers to debug as well, for example:

category.UnifiedSearch=DEBUG
category.IndexScopedSearch=DEBUG

This change takes effect immediately for all searches started after such changes.

Debug Splunk Web

Change the logging level for the splunkweb process by editing the file:

$SPLUNK_HOME/etc/log.cfg 
or if you have created your own
$SPLUNK_HOME/etc/log-local.cfg

Locate the [python] stanza and change the contents to:

[python]
splunk = DEBUG
# other lines should be removed

The logging component names are hierarchical so setting the top level splunk component will affect all loggers unless a more specific setting is provided, like splunk.search = INFO.

Restart the splunkweb process with the command ./splunk restart splunkweb. The additional messages are output in the file $SPLUNK_HOME/var/log/splunk/web_service.log.

PREVIOUS
What Splunk logs about itself
  NEXT
About metrics.log

This documentation applies to the following versions of Splunk® Enterprise: 4.3, 4.3.1, 4.3.2, 4.3.3


Was this documentation topic helpful?

Enter your email address, and someone from the documentation team will respond to you:

Please provide your comments here. Ask a question or make a suggestion.

You must be logged into splunk.com in order to post comments. Log in now.

Please try to keep this discussion focused on the content covered in this documentation topic. If you have a more general question about Splunk functionality or are experiencing a difficulty with Splunk, consider posting a question to Splunkbase Answers.

0 out of 1000 Characters