What Splunk logs about itself

What Splunk logs about itself

Splunk keeps track of its activity by logging to various files in $SPLUNK_HOME/var/log/splunk.

Splunk's internal log files are rolled based on size. You can change the default log rotation size by editing $SPLUNK_HOME/etc/log.cfg.

Search these files in Splunk Web by typing:

Internal logs

Here is a complete list with descriptions of the internal logs in $SPLUNK_HOME/var/log/splunk. Splunk's internal logs are useful for troubleshooting or metric analysis.

audit.log

Log of audit events.

crawl.log

Log of crawl activities.

inputs.log

license_audit.log

Continuous audit of license violations.

metrics.log

Contains information about CPU usage and Splunk's data processing. The metrics.log file is a sampling of the top ten items in each category for in 30-second intervals, based on the size of _raw. It can be used for limited analysis of volume trends for data inputs. For more information about what's in metrics.log, refer to Work with metrics.log as well as this developer blog post about Splunk forwarder and indexer metrics.

migration.log

A log of events during install and migration. Specifies which files were altered during upgrade.

python.log

A log of python events within Splunk. Useful for debugging REST endpoints and communication with splunkd.

scheduler.log

A log of all actions (successful or unsuccessful) performed by the splunkd scheduler. Typically, this will show scheduled search activity.

searchhistory.log

A log of all searches performed on the server since installation or the most recent splunk clean command.

splunkd_stdout.log

The Unix standard output device for the server.

splunkd_stderr.log

The Unix standard error device for the server.

splunklogger.log

A subset of the Splunk server's own log events since installation or the most recent splunk clean command. This file is sent to index::splunklogger and can be searched through Splunk Web.

splunkd.log

A record of actions made by the Splunk server. May be requested by Splunk Support for troubleshooting purposes. Any stderr messages generated by scripted inputs, scripted search commands, and so on, are logged here.

splunkmon.log

Log of Splunk's watchdog process. Note: The watchdog process and command are deprecated and should not be used.

web_access.log

A record of actions made by Splunk Web, in an Apache access_log format.

web_service.log

A record of actions made by Splunk Web.

Debug mode

Splunk has a debugging parameter (--debug) you can use when starting Splunk from the CLI. Navigate to Splunk's CLI $SPLUNK_HOME/bin and use the ./splunk command:

./splunk start --debug 

This command outputs logs to $SPLUNK_HOME/var/log/splunk/splunkd.log. To turn off debugging, stop or restart Splunk.

Note: Running Splunk with debugging turned on outputs a large amount of information. Make sure you do not leave debugging on for any significant length of time.

log.cfg

For more granular debugging messages, you can change log levels by editing $SPLUNK_HOME/etc/log.cfg. This affects Splunk's internal logs.

You can change the following categories in log.cfg. Set the category you wish to debug from WARN or INFO to DEBUG.

The message levels, in order from least to most urgent are:

• DEBUG
• INFO
• WARN
• ERROR
• FATAL
• CRIT

rootCategory=WARN,A1
category.LicenseManager=INFO
category.TcpOutputProc=INFO
category.TcpInputProc=INFO
category.UDPInputProcessor=INFO
category.SavedSplunker=INFO
category.DistributedMgr=INFO
category.DistributedExecutionContext=INFO
category.DistributedDeploymentProcessor=INFO
category.DistributedDeploymentClientProcessor=INFO
category.DistributedDeploymentClientMgr=INFO
category.DistributedDeploymentMgr=INFO
category.ThruputProcessor=WARN
category.ShutdownHandler=WARN
# leave loader at INFO!  this is what gives us our build + system info...
category.loader=INFO
category.ulimit=INFO
category.SearchPerformance=INFO
category.SearchPipelinePerformance=WARN

To change the maximum size of a log file before it rolls, change the maxFileSize value (in bytes) for the desired file:

appender.A1=RollingFileAppender
appender.A1.fileName=${SPLUNK_HOME}/var/log/splunk/splunkd.log
appender.A1.maxFileSize=250000000
appender.A1.maxBackupIndex=5
appender.A1.layout=PatternLayout
appender.A1.layout.ConversionPattern=%d{%m-%d-%Y %H:%M:%S.%l} %-5p %c - %m%n

If you change this file, restart Splunk.

log-local.cfg

You can put log.cfg settings into a local file, log-local.cfg file, residing in the same directory as log.cfg. The settings in log-local.cfg take precedence. And unlike log.cfg, the log-local.cfg file doesn't get overwritten on upgrade.

Use Splunk Web to manage logs

To view and manage logs, you can use the Manager:

1. Navigate to Manager > System settings > System logging. This generates a list of log channels and their status.

2. To change the logging level for a particular log channel, click on that channel. This brings up a page specific to that channel.

3. On the log channel's page, you can change its logging level.

When you change the logging level, note the following:

• The change is immediate and dynamic.
• The change is not persistent; it goes away when Splunk is restarted.

Important: Manager > System settings > System logging is meant only for dynamic and temporary changes to Splunk log files. For permanent changes, use $SPLUNK_HOME/etc/log.cfg instead.

Search logs

If you have any long-running real-time searches, you might want to adjust the maximum size of your search logs.

Search logs are located in $SPLUNK_HOME/var/run/splunk/dispatch/rt_<job_id>/. These logs are rotated when they reach a default maximum size of 25MB. Splunk keeps up to five of them, so the total log size for a search can conceivably grow as large as 125MB.

Most searches are unlikely to generate logs anywhere near 25MB in size; however, it can become an issue if you have ongoing real-time searches.

To adjust the log size, edit $SPLUNK_HOME/etc/log-searchprocess.cfg.

• Was this topic useful?
• Post a comment