What Splunk logs about itself
This documentation does not apply to the most recent version of Splunk. Click here for the latest version.
Contents
What Splunk logs about itself
Splunk keeps track of its activity by logging to various files in $SPLUNK_HOME/var/log/splunk.
Splunk's internal log files are rolled based on size. You can change the default log rotation size by editing $SPLUNK_HOME/etc/log.cfg.
Search these files in Splunk Web by typing:
index=_internalInternal logs
Here is a list with descriptions of the internal logs in $SPLUNK_HOME/var/log/splunk. Splunk's internal logs are useful for troubleshooting or metric analysis.
audit.log
Log of audit events.
btool.log
Brief information on the past execution of the btool Splunk sub-command.
crawl.log
Log of crawl activities.
first_install.log
A record of the version and architecture used for the instance's first installation.
license_audit.log
Audit of license usage (daily and on restart) and violations.
metrics.log
Contains information about CPU usage and Splunk's data processing. The metrics.log file is a sampling of the top ten items in each category in user-configurable 30-second intervals, based on the size of _raw. It can be used for limited analysis of volume trends for data inputs. For more information about what's in metrics.log, refer to Work with metrics.log as well as this developer blog post about Splunk forwarder and indexer metrics.
migration.log.*
A log of events during install and migration. Specifies which files were altered during upgrade.
python.log
A log of python script executions and behavior within Splunk. Useful for debugging REST endpoints and communication with splunkd.
scheduler.log
A log of all actions (successful or unsuccessful) performed by the splunkd scheduler. Typically, this will show scheduled search activity.
searches.log
A log of all searches performed on the server since installation or the most recent splunk clean command.
splunkd_stdout.log
Text written to Unix standard output by the splunkd daemon.
splunkd_stderr.log
Text written to Unix standard error by the splunkd daemon. Very useful to review for crash situations.
splunkd.log
A log of actions and events recorded by splunkd. May be requested by Splunk Support for troubleshooting purposes.
splunkd_access.log
A record of REST API calls served by splunkd, in an Apache access_log format.
web_access.log
A record of pages/assets served by Splunk Web, in an Apache access_log format.
web_service.log
A record of actions and events recorded by Splunk Web.
Debug mode
Splunk has a debugging parameter (--debug) you can use when starting Splunk from the CLI. Navigate to Splunk's CLI $SPLUNK_HOME/bin and use the ./splunk command:
./splunk start --debug
This command outputs logs to $SPLUNK_HOME/var/log/splunk/splunkd.log. To turn off debugging, stop or restart Splunk.
Note: Running Splunk with debugging turned on outputs a large amount of information. Make sure you do not leave debugging on for any significant length of time.
log.cfg
For more granular debugging messages, you can change log levels by editing $SPLUNK_HOME/etc/log.cfg. This affects Splunk's internal logs.
You can change the following categories in log.cfg. Set the category you wish to debug from WARN or INFO to DEBUG.
The message levels, in order from least to most urgent are:
- DEBUG
- INFO
- WARN
- ERROR
- FATAL
- CRIT
rootCategory=WARN,A1 category.LicenseManager=INFO category.TcpOutputProc=INFO category.TcpInputProc=INFO category.UDPInputProcessor=INFO category.SavedSplunker=INFO category.DistributedMgr=INFO category.DistributedExecutionContext=INFO category.DistributedDeploymentProcessor=INFO category.DistributedDeploymentClientProcessor=INFO category.DistributedDeploymentClientMgr=INFO category.DistributedDeploymentMgr=INFO category.ThruputProcessor=WARN category.ShutdownHandler=WARN # leave loader at INFO! this is what gives us our build + system info... category.loader=INFO category.ulimit=INFO category.SearchPerformance=INFO category.SearchPipelinePerformance=WARN
To change the maximum size of a log file before it rolls, change the maxFileSize value (in bytes) for the desired file:
appender.A1=RollingFileAppender
appender.A1.fileName=${SPLUNK_HOME}/var/log/splunk/splunkd.log
appender.A1.maxFileSize=250000000
appender.A1.maxBackupIndex=5
appender.A1.layout=PatternLayout
appender.A1.layout.ConversionPattern=%d{%m-%d-%Y %H:%M:%S.%l} %-5p %c - %m%n
If you change this file, restart Splunk.
log-local.cfg
You can put log.cfg settings into a local file, log-local.cfg file, residing in the same directory as log.cfg. The settings in log-local.cfg take precedence. And unlike log.cfg, the log-local.cfg file doesn't get overwritten on upgrade.
Use Splunk Web to manage logs
To view and manage logs, you can use the Manager:
1. Navigate to Manager > System settings > System logging. This generates a list of log channels and their status.
2. To change the logging level for a particular log channel, click on that channel. This brings up a page specific to that channel.
3. On the log channel's page, you can change its logging level.
When you change the logging level, note the following:
- The change is immediate and dynamic.
- The change is not persistent; it goes away when Splunk is restarted.
Important: Manager > System settings > System logging is meant only for dynamic and temporary changes to Splunk log files. For permanent changes, use $SPLUNK_HOME/etc/log.cfg instead.
This documentation applies to the following versions of Splunk: 4.1 , 4.1.1 , 4.1.2 , 4.1.3 , 4.1.4 , 4.1.5 , 4.1.6 , 4.1.7 , 4.1.8 View the Article History for its revisions.
Comments
I created a log-local.cfg and restarted splunk. When can I expect the log files to change?
Rich
splunklogger.log - A subset of the Splunk server's own log events since installation or the most recent splunk clean command. This file is sent to index::splunklogger and can be searched through Splunk Web.
I believe this statement is no longer valid as of 4.1.4??