What Splunk software logs about itself

Splunk software is capable of many tasks, from ingesting data, processing data into events, indexing events, and searching those events. All of these tasks, and many of the steps in-between, generate data that the Splunk software records into log files.

Logging locations

The Splunk software internal logs are located in: $SPLUNK_HOME/var/log/splunk. This path is monitored by default, and the contents are sent to the _internal index. If the Splunk software is configured as a Forwarder, a subset of the logs are monitored and sent to the indexing tier.

The Splunk Introspection logs are located in $SPLUNK_HOME/var/log/introspection. These logs record data about the impact of the Splunk software on the host system. This path is monitored by default, and the contents are sent to the _introspection index. If the Splunk software is configured as a Forwarder, the monitored logs are sent to the indexing tier. See About Splunk Enterprise platform instrumentation.

The Splunk search logs are located in sub-folders under $SPLUNK_HOME/var/run/splunk/dispatch/. These logs record data about a search, including run time and other performance metrics. The search logs are not indexed by default. See Dispatch directory and search artifacts in the Search Manual.

Internal logs

A list of the internal logs in $SPLUNK_HOME/var/log/splunk with descriptions of their use.

Log file name	Useful for?
audit.log	Information about user activities such as a failed or successful user log in, modifying a setting, updating a lookup file, or running a search. For example, if you're looking for information about a saved search, audit.log matches the name of a saved search (savedsearch_name) with its search ID (search_id), user, and time fields. With the search_id, you can review the logs of a specific search in the search dispatch directory. See search dispatch directory in the Search Manual and audit events in the Securing Splunk Manual. Audit.log is the only log indexed to the _audit index.
btool.log	A log of btool activity. See btool.
conf.log	Contains messages about configuration replication related to Search Head Clustering. See search head clustering in the Distributed Search manual.
crawl.log	No longer used.
export_metrics.log	Log of Hadoop Connect metrics. See metrics related to exporting data with Hadoop Connect in the Deploy and Use Splunk Hadoop Connect manual.
first_install.log	Shows version number.
http_event_collector_metrics.log	HTTP Event Collector saves metrics about itself to this log file. See Troubleshoot HTTP Event Collector in the Getting Data In manual.
inputs.log	No longer used.
intentions.log	No longer used.
kvstore.log	Log of metrics for KV store.
license_audit.log	No longer used.
license_usage.log	Indexed volume in bytes per pool, index, source, source type, and host. Available only on a Splunk instance configured as a license master.
license_usage_summary.log	Daily indexed volume in bytes per pool, stack, and host. Available only on a Splunk instance configured as a license master. The log in indexed into _telemetry. See Share data in Splunk Enterprise in the Admin Manual.
metrics.log	Contains periodic snapshots of Splunk performance and system data, including information about CPU usage by internal processors and queue usage in Splunk's data processing. The metrics.log file is a sampling of the top ten items in each category in 30 second intervals, based on the size of _raw. It can be used for limited analysis of volume trends for data inputs. See About metrics.log and Work with metrics.log.
migration.log	A log of events during install and migration. Specifies which files were altered during upgrade.
mongod.log	Contains runtime messages from the Splunk Enterprise KVStore. See App key value store in the Admin Manual.
python.log	Python events within Splunk. Useful for debugging REST endpoints, communication with splunkd, PDF Report Server App, Splunk Web display issues, sendmail (email alerts), and scripted or modular inputs. This log records "WARNING" instead of "WARN" for second most verbose logging level. The python.log is unmanaged by the Splunk platform. To manage the log file rotation, use an external log management service.
remote_searches.log	Messages from StreamedSearch channel. This code is executed on the search peers when a search head makes a search request. This file contains useful information on indexers regarding searches they're participating in.
scheduler.log	All actions (successful or unsuccessful) performed by the splunkd search and alert scheduler. Typically, this shows scheduled search activity.
search_messages.log	A digest of any critical messages recorded in the info.csv of all dispatched searches. The log is updated when DispatchReaper reaps the dispatch directories. Enabled by default. See limits.conf in the Admin Manual.
searches.log	No longer used. Instead, use the following search syntax: | history. This shows all the searches that have been run, plus stats for the searches.
searchhistory.log	No longer used.
splunkd.log	The primary log for the Splunk server. The log is often requested by Splunk Support for troubleshooting purposes. In addition, any stderr messages generated by scripted inputs, scripted search commands, and similar are logged here.
splunkd_access.log	Any action done from splunkd through the UI is logged here, including splunkweb, the CLI, all POST GET actions, deleted saved searches, and other programs accessing the REST endpoints. Also logs the time taken to respond to the requests. Search job artifacts logged here include size of data returned with search.
splunkd_stderr.log	The Unix standard error device for the server. Typically this contains (for *nix) times of healthy start and stop events, as well as various errors like exceptions, assertions, and errors generated by libraries and the operating system.
splunkd_stdout.log	The Unix standard output device for the server.
splunkd_ui_access.log	Starting in 6.2, contains a significant portion of the types of events that used to be logged in web_access.log.
splunkd-utility.log	This log is written to by the prereq-checking utils splunkd clone-prep-clear-config, splunkd validatedb, splunkd check-license, splunkd check-transforms-keys, and splunkd rest (for offline CLI). Each util logs Splunk version, some basic config, and current OS limits like max number of threads, and then messages specific to the util. Consult this log file when splunkd didn't start.
web_access.log	Requests made of Splunk Web, in an Apache access_log format. Much of the types of events logged here are logged in splunkd_ui_access.log starting in 6.2.
web_service.log	Primary log written by splunkweb. Records actions made by splunkweb. Note: the log records "WARNING" instead of "WARN" for second most verbose logging level.

Some log files are not created until your Splunk instance uses them. Other logs are created, but will remain empty until events are written.

The log management process

The internal logs are rolled based on file size, with a number of historical logs kept. The historical rotation for most internal logs is 5 files of 25MB each. You can review the log rotation settings in $SPLUNK_HOME/etc/log.cfg.

For long-term changes to the log management process, such as increasing the historical log rotation or log size, we recommend creating a $SPLUNK_HOME/etc/log-local.cfg file and placing your changes in there. The settings in log-local.cfg take precedence over log.cfg, and the file does not get overwritten on upgrade.

Logging levels

Splunk platform internal logging levels are DEBUG INFO WARN ERROR FATAL from most to least verbose. The debug logging is disabled by default. See enabling debug logging.

Use Splunk Web to manage logging-level

To change the logging-level using Splunk Web:

1. Navigate to Settings > Server settings > Server logging. This generates a list of log channels and their status.

2. To change the logging level for a particular log channel, click on that channel. This brings up a page specific to that channel.

3. On the log channel's page, change the logging level.

When you change the logging level, note the following:

• The change is immediate and dynamic.
• The change is not persistent; it goes away when the Splunk service is restarted.

Searching internal logs

By default, only the Admin role can search the _internal index, and the _internal index must be called explicitly. Search the internal log files in Splunk Web by typing:

index=_internal

Search for errors and warnings by typing:

index=_internal (log_level=error OR log_level=warn*)

Search the internal logs using Pivot

Splunk Enterprise includes data models constructed from the internal logs. To access the internal log data models, in the Search & Reporting app in Splunk Web, click Datasets.

Set logging levels and log channels for a search

You can use the noop command to set the logging level and logging channel for a specific search job. Use the log_<level> argument to identify a logging level and one or more logging channels. For more information see noop in the Search Reference Manual.