Enable debug logging
The Splunk platform internal logging levels are as follows, from most to least verbose:
- DEBUG
- INFO
- WARN
- ERROR
- FATAL
This topic gives a few popular options for how you might want to put Splunk into debug mode.
Splunk software debug mode is very detailed. The verbosity that debug mode creates might obscure something that could have helped you diagnose your problem. Running Splunk software in debug mode for an extended time can generate very large internal log files. Except where necessary, do not run production systems in debug mode.
Enable debug logging on all of splunkd.log
Splunk software has a debugging parameter (--debug
) that you can use when you start Splunk software from the CLI in *nix. This command outputs logs to the $SPLUNK_HOME/var/log/splunk/splunkd.log
file.
This option is not available on Windows. To enable debugging on Splunk software running on Windows, enable debugging on a specific processor. See Enable debug logging in Splunk Web or Enable debug logging using log.cfg.
- Navigate to
$SPLUNK_HOME/bin
. - Stop the Splunk platform instance, if it is running.
- Save your existing
splunkd.log
file by renaming it, likesplunkd.log.old
. - Restart the instance in debug mode with
splunk start --debug
. - When you notice the problem, stop the instance.
- Move the new
splunkd.log
file elsewhere and restore the old file. - Stop or restart the instance normally (without the
--debug
flag) to disable debug logging.
Not all messages marked WARN or ERROR indicate actual problems with Splunk software; some indicate that a feature is not being used.
Enable debug logging for a specific processor within splunkd.log
Specific areas of debug logging can be enabled to collect debugging details over a longer period with minimal performance impact. See the category settings in the file $SPLUNK_HOME/etc/log-local.cfg
to set specific log levels without enabling a large number of categories as with --debug
. Restart the Splunk platform instance after changing this file.
If you upgrade the Splunk platform instance, the upgrade overwrites any changes to $SPLUNK_HOME/etc/log.cfg
.
In Splunk Web
You can enable these DEBUG settings via Splunk Web if you have admin privileges.
- Navigate to Settings > Server settings > Server logging.
- Search for the processor names using the text box.
- Click on the processor name to change the logging level to DEBUG.
You do not need to restart Splunk. In fact, these changes will not persist if you restart the Splunk instance.
In log.cfg
If you want the processors to be in DEBUG on startup, or if you want to turn on debugging for a few processors, create or edit a $SPLUNK_HOME/etc/log-local.cfg
file to override changes in $SPLUNK_HOME/etc/log.cfg
.
You can use the category.AuditLogger
setting to increase or decrease the logging leave for audit events. By default, the Splunk platform logs these events at the DEBUG level, down from the INFO level.
- In $SPLUNK_HOME/etc/log.cfg, find the category.* entry that relates to the processor you are interested in.
- Copy that line to log-local.cfg with INFO or WARN modified to DEBUG.
- Save the file and close it.
- Restart the Splunk platform instance.
There will not always be an existing entry for the processor you are interested in, and it may take some digging through the logs or documentation to find the correct processor.
For example, to see how often Splunk software is updating or retrieving progress-tracking records for a particular file, put 'FileInputTracker' in DEBUG. Update the existing entry to read
category.FileInputTracker=DEBUG
Or for investigating problems monitoring files, use the FileInputTracker and selectProcessor categories.
After you make the changes every time Splunk software checks the inputs file, it will be recorded in $SPLUNK_HOME/var/log/splunk/splunkd.log.
Remember to change the settings that you made back when you are finished investigating.
If a default level is not specified for a category, the logging level defaults to your rootCategory setting.
Do not make changes to the category.loader
setting. This setting provides build and system information about the instance.
To change the maximum size of a log file before it rolls, change the maxFileSize
value (in bytes) for the desired file:
appender.A1=RollingFileAppender appender.A1.fileName=${SPLUNK_HOME}/var/log/splunk/splunkd.log appender.A1.maxFileSize=25000000 appender.A1.maxBackupIndex=5 appender.A1.layout=PatternLayout appender.A1.layout.ConversionPattern=%d{%m-%d-%Y %H:%M:%S.%l} %-5p %c - %m%n
About precedence
If you have duplicate lines in log.cfg, the last line takes precedence. For example,
category.databasePartitionPolicy=INFO category.databasePartitionPolicy=DEBUG
sets the databasePartitionPolicy
category to DEBUG.
The other log-*.cfg files behave similarly when you add categories. To set only some things in a search.log into debug mode, in log-searchprocess.cfg add a new category line after the rootCategory:
rootCategory=INFO,searchprocessAppender category.<foo>=DEBUG appender.searchprocessAppender=RollingFileAppender
This leaves everything else as it was, which means only the debug messages you want are generated. Putting rootCategory into DEBUG mode makes the dispatch directories huge, so it is not a good choice for long-running debug.
log-local.cfg
You can put log.cfg
settings into a local file, log-local.cfg
file, residing in the same directory as log.cfg
. The settings in log-local.cfg
take precedence. And unlike log.cfg
, the log-local.cfg
file doesn't get overwritten on upgrade.
With endpoints
You can access a debugging endpoint that shows status information about monitored files:
https://your-splunk-server:8089/services/admin/inputstatus/TailingProcessor:FileStatus
Enable debug messages from the CLI
./splunk _internal call /services/server/logger/TailingProcessor -post:level DEBUG
This search returns the message "HTTP Status: 200". This is not an error and is normal.
You can also enable debugging with this command:
./splunk set log-level TailingProcessor -level DEBUG
Enable debug logging for search processes
Search processes obey the etc/log-searchprocess.cfg rules. Similar to splunkd, they can be overridden in etc/log-searchprocess-local.cfg.
All loggers can be set to DEBUG by adding a line such as
rootCategory=DEBUG,searchprocessAppender
Specific loggers can be set to debug as well, for example:
category.UnifiedSearch=DEBUG category.IndexScopedSearch=DEBUG
This change takes effect immediately for all searches started after the change.
Use the noop command to change appender attributes for a search
You can use the log_appender
attribute of the noop
command to change the attributes for a log-searchprocess.log
appender for a specific run of a search.
See noop
in the Search Reference.
Debug Splunk Web
Change the logging level for the splunkweb process by editing the file:
$SPLUNK_HOME/etc/log.cfg
or if you have created your own
$SPLUNK_HOME/etc/log-local.cfg
Locate the [python]
stanza and change the contents to:
[python] splunk = DEBUG # other lines should be removed
The logging component names are hierarchical so setting the top level splunk
component will affect all loggers unless a more specific setting is provided, like splunk.search = INFO
.
Restart the splunkweb process with the command ./splunk restart splunkweb
. The additional messages are output in the file $SPLUNK_HOME/var/log/splunk/web_service.log
.
What Splunk software logs about itself | About metrics.log |
This documentation applies to the following versions of Splunk® Enterprise: 8.1.0, 8.1.1, 8.1.2, 8.1.3, 8.1.4, 8.1.5, 8.1.6, 8.1.7, 8.1.8, 8.1.9, 8.1.10, 8.1.11, 8.1.12, 8.1.13, 8.1.14, 8.2.0, 8.2.1, 8.2.2, 8.2.3, 8.2.4, 8.2.5, 8.2.6, 8.2.7, 8.2.8, 8.2.9, 8.2.10, 8.2.11, 8.2.12, 9.0.0, 9.0.1, 9.0.2, 9.0.3, 9.0.4, 9.0.5, 9.0.6, 9.0.7, 9.0.8, 9.0.9, 9.0.10, 9.1.0, 9.1.1, 9.1.2, 9.1.3, 9.1.4, 9.1.5, 9.1.6, 9.2.0, 9.2.1, 9.2.2, 9.2.3, 9.3.0, 9.3.1
Feedback submitted, thanks!