Splunk log files
This documentation does not apply to the most recent version of Splunk. Click here for the latest version.
Contents
Splunk log files
Splunk keeps track of its activity by logging to various files in $SPLUNK_HOME/var/log/splunk.
Splunk's internal log files are rolled based on size. You can change the default log rotation size by editing $SPLUNK_HOME/etc/log.cfg.
Search these files in Splunk Web by typing:
index::_internalInternal logs
Here is a complete list with descriptions of the internal logs in $SPLUNK_HOME/var/log/splunk. Splunk's internal logs are useful for troubleshooting or metric analysis.
audit.log
Log of audit events.
crawl.log
Log of crawl activities.
inputs.log
license_audit.log
Continuous audit of license violations.
metrics.log
Contains information about CPU usage and Splunk's data processing. The metrics.log file is a sampling of the top ten items in each category for in 30-second intervals, based on the size of _raw. It can be used for limited analysis of volume trends for data inputs. For more information about what's in metrics.log, refer to Work with metrics.log as well as this developer blog post about Splunk forwarder and indexer metrics.
migration.log
A log of events during install and migration. Specifies which files were altered during upgrade.
python.log
A log of python events within Splunk. Useful for debugging REST endpoints and communication with splunkd.
searchhistory.log
A log of all searches performed on the server since installation or the most recent splunk clean command.
splunkd_stdout.log
The Unix standard output device for the server.
splunkd_stderr.log
The Unix standard error device for the server.
splunklogger.log
A subset of the Splunk server's own log events since installation or the most recent splunk clean command. This file is sent to index::splunklogger and can be searched through Splunk Web.
splunkd.log
A record of actions made by the Splunk server. May be requested by Splunk Support for troubleshooting purposes.
splunkmon.log
Log of splunk's watchdog process. Note: The watchdog process and command are deprecated and should not be used.
web_access.log
A record of actions made by Splunk Web, in an Apache access_log format.
web_service.log
A record of actions made by Splunk Web.
debug
Splunk has a debugging parameter (--debug) you can add when starting Splunk from the CLI (with ./splunk start).
./splunk start --debug
Note: Navigate to Splunk's CLI $SPLUNK_HOME/bin and use the ./splunk command. You can also add Splunk to your path.
This command outputs logs to $SPLUNK_HOME/var/log/splunk/splunkd.log. To turn off debugging, stop or restart Splunk.
Note: running Splunk with debugging turned on outputs a large amount of information. Make sure you do not leave debugging on for any significant length of time.
To dynamically enable debugging messages for a particular category, you can use these searches:
From the UI:
| debug cmd=logchange param1=FileInputTracker param2=DEBUG
| debug cmd=logchange param1=selectProcessor param2=DEBUG
You will get a message "Error in 'DebugCommand'..." This is normal and can be ignored.
From the CLI:
$ ./splunk search "| debug cmd=logchange param1=FileInputTracker param2=DEBUG" -auth admin:changeme
You will get the message "FATAL: Error in 'DebugCommand': Setting priority of..." This is normal and can be ignored.
The log.cfg file is not changed and the original settings from this file will be restored on next restart.
log.cfg
For more granular debugging messages, you can change log levels by editing $SPLUNK_HOME/etc/log.cfg. This affects Splunk's internal logs.
You can change the following categories in log.cfg. Set the category you wish to debug from WARN or INFO to DEBUG.
The message levels, in order from least to most urgent are:
- DEBUG
- INFO
- WARN
- ERROR
- FATAL
- CRIT
rootCategory=WARN,A1 category.LicenseManager=INFO category.TcpOutputProc=INFO category.TcpInputProc=INFO category.UDPInputProcessor=INFO category.SavedSplunker=INFO category.DistributedMgr=INFO category.DistributedExecutionContext=INFO category.DistributedDeploymentProcessor=INFO category.DistributedDeploymentClientProcessor=INFO category.DistributedDeploymentClientMgr=INFO category.DistributedDeploymentMgr=INFO category.ThruputProcessor=WARN category.ShutdownHandler=WARN # leave loader at INFO! this is what gives us our build + system info... category.loader=INFO category.ulimit=INFO category.SearchPerformance=INFO category.SearchPipelinePerformance=WARN
To change the maximum size of a log file before it rolls, change the maxFileSize value (in bytes) for the desired file:
appender.A1=RollingFileAppender
appender.A1.fileName=${SPLUNK_HOME}/var/log/splunk/splunkd.log
appender.A1.maxFileSize=250000000
appender.A1.maxBackupIndex=5
appender.A1.layout=PatternLayout
appender.A1.layout.ConversionPattern=%d{%m-%d-%Y %H:%M:%S.%l} %-5p %c - %m%n
If you modify logging settings with a logchange search or through the UI, these are not persisted in log.cfg. Your original settings will be restored on the next restart.
log-local.cfg
You can put log.cfg settings into a local file, log-local.cfg file, residing in the same directory as log.cfg. The settings in log-local.cfg take precedence. And unlike log.cfg, the log-local.cfg file doesn't get overwritten on upgrade.
This documentation applies to the following versions of Splunk: 3.3 , 3.3.1 , 3.3.2 , 3.3.3 , 3.3.4 , 3.4 , 3.4.1 , 3.4.2 , 3.4.3 , 3.4.5 , 3.4.6 , 3.4.8 , 3.4.9 , 3.4.10 , 3.4.11 , 3.4.12 , 3.4.13 , 3.4.14 , 4.0 , 4.0.1 , 4.0.2 , 4.0.3 , 4.0.4 , 4.0.5 , 4.0.6 , 4.0.7 , 4.0.8 , 4.0.9 , 4.0.10 , 4.0.11 View the Article History for its revisions.