What Splunk software logs about itself
Splunk software is capable of many tasks, from ingesting data, processing data into events, indexing events, and searching those events. All of these tasks, and many of the steps in-between, generate data that the Splunk software records into log files.
The Splunk software internal logs are located in:
$SPLUNK_HOME/var/log/splunk. This path is monitored by default, and the contents are sent to the _internal index. If the Splunk software is configured as a Forwarder, a subset of the logs are monitored and sent to the indexing tier.
The Splunk Introspection logs are located in
$SPLUNK_HOME/var/log/introspection. These logs record data about the impact of the Splunk software on the host system. This path is monitored by default, and the contents are sent to the _introspection index. If the Splunk software is configured as a Forwarder, the monitored logs are sent to the indexing tier. See About Splunk Enterprise platform instrumentation.
The Splunk search logs are located in sub-folders under
$SPLUNK_HOME/var/run/splunk/dispatch/. These logs record data about a search, including run time and other performance metrics. The search logs are not indexed by default. See Dispatch directory and search artifacts in the Search Manual.
A list of the internal logs in
$SPLUNK_HOME/var/log/splunk with descriptions of their use.
|Log file name||Useful for?|
|audit.log||Information about user activities such as a failed or successful user log in, modifying a setting, updating a lookup file, or running a search. For example, if you're looking for information about a saved search, audit.log matches the name of a saved search (savedsearch_name) with its search ID (search_id), user, and time fields. With the search_id, you can review the logs of a specific search in the search dispatch directory. See search dispatch directory in the Search Manual and audit events in the Securing Splunk Manual. Audit.log is the only log indexed to the |
|btool.log||A log of btool activity. See btool.|
|conf.log||Contains messages about configuration replication related to Search Head Clustering. See search head clustering in the Distributed Search manual.|
|configuration_change.log||Tracks changes to .conf files at the filesystem level, including the creation of .conf files in the monitored file paths. |
Configuration change monitoring is enabled by default.
|export_metrics.log||Log of Hadoop Connect metrics.|
|first_install.log||Shows version number.|
|http_event_collector_metrics.log||HTTP Event Collector saves metrics about itself to this log file. See Troubleshoot HTTP Event Collector in the Getting Data In manual.|
|kvstore.log||Log of metrics for KV store.|
|license_usage.log||Indexed volume in bytes per pool, index, source, source type, and host. Available only on a Splunk instance configured as a license master.|
|license_usage_summary.log||Daily indexed volume in bytes per pool, stack, and host. Available only on a Splunk instance configured as a license master. The log in indexed into _telemetry. See Share data in Splunk Enterprise in the Admin Manual.|
|metrics.log||Contains periodic snapshots of Splunk performance and system data, including information about CPU usage by internal processors and queue usage in Splunk's data processing. The metrics.log file is a sampling of the top ten items in each category in 30 second intervals, based on the size of _raw. It can be used for limited analysis of volume trends for data inputs. See About metrics.log and Work with metrics.log.|
|migration.log||A log of events during install and migration. Specifies which files were altered during upgrade.|
|mongod.log||Contains runtime messages from the Splunk Enterprise KVStore. See App key value store in the Admin Manual.|
|python.log||Python events within Splunk. Useful for debugging REST endpoints, communication with splunkd, PDF Report Server App, Splunk Web display issues, sendmail (email alerts), and scripted or modular inputs. This log records "WARNING" instead of "WARN" for second most verbose logging level.
|remote_searches.log||Messages from StreamedSearch channel. This code is executed on the search peers when a search head makes a search request. This file contains useful information on indexers regarding searches they're participating in.|
|scheduler.log||All actions (successful or unsuccessful) performed by the splunkd search and alert scheduler. Typically, this shows scheduled search activity.|
|search_messages.log||A digest of any critical messages recorded in the info.csv of all dispatched searches. The log is updated when DispatchReaper reaps the dispatch directories. Enabled by default. See limits.conf in the Admin Manual.|
|searches.log||No longer used. Instead, use the following search syntax: |
|splunkd.log||The primary log for the Splunk server. The log is often requested by Splunk Support for troubleshooting purposes. In addition, any |
|splunkd_access.log||Any action done from splunkd through the UI is logged here, including splunkweb, the CLI, all POST GET actions, deleted saved searches, and other programs accessing the REST endpoints. Also logs the time taken to respond to the requests. Search job artifacts logged here include size of data returned with search.|
|splunkd_stderr.log||The Unix standard error device for the server. Typically this contains (for *nix) times of healthy start and stop events, as well as various errors like exceptions, assertions, and errors generated by libraries and the operating system.|
|splunkd_stdout.log||The Unix standard output device for the server.|
|splunkd_ui_access.log||Starting in 6.2, contains a significant portion of the types of events that used to be logged in web_access.log.|
|splunkd-utility.log||This log is written to by the prereq-checking utils |
|web_access.log||Requests made of Splunk Web, in an Apache access_log format. Much of the types of events logged here are logged in splunkd_ui_access.log starting in 6.2.|
|web_service.log||Primary log written by splunkweb. Records actions made by splunkweb. Note: the log records "WARNING" instead of "WARN" for second most verbose logging level.|
Some log files are not created until your Splunk instance uses them. Other logs are created, but will remain empty until events are written.
The log management process
The internal logs are rolled based on file size, with a number of historical logs kept. The historical rotation for most internal logs is 5 files of 25MB each. You can review the log rotation settings in
For long-term changes to the log management process, such as increasing the historical log rotation or log size, we recommend creating a
$SPLUNK_HOME/etc/log-local.cfg file and placing your changes in there. The settings in
log-local.cfg take precedence over
log.cfg, and the file does not get overwritten on upgrade.
Splunk platform internal logging levels are
DEBUG INFO WARN ERROR FATAL from most to least verbose. The debug logging is disabled by default. See enabling debug logging.
Use Splunk Web to manage logging-level
To change the logging-level using Splunk Web:
1. Navigate to Settings > Server settings > Server logging. This generates a list of log channels and their status.
2. To change the logging level for a particular log channel, click on that channel. This brings up a page specific to that channel.
3. On the log channel's page, change the logging level.
When you change the logging level, note the following:
- The change is immediate and dynamic.
- The change is not persistent; it goes away when the Splunk service is restarted.
Searching internal logs
By default, only the Admin role can search the _internal index, and the _internal index must be called explicitly. Search the internal log files in Splunk Web by typing:
Search for errors and warnings by typing:
index=_internal (log_level=error OR log_level=warn*)
Search the internal logs using Pivot
Splunk Enterprise includes data models constructed from the internal logs. To access the internal log data models, in the Search & Reporting app in Splunk Web, click Datasets.
Set logging levels and log channels for a search
You can use the
noop command to set the logging level and logging channel for a specific search job. Use the
log_<level> argument to identify a logging level and one or more logging channels.
For more information see
noop in the Search Reference Manual.
Use btool to troubleshoot configurations
Enable debug logging
This documentation applies to the following versions of Splunk® Enterprise: 8.2.0, 8.2.1, 8.2.2, 8.2.3, 8.2.4, 8.2.5, 8.2.6, 8.2.7