What Splunk software logs about itself

Splunk software is capable of many tasks, from ingesting data, processing data into events, indexing events, and searching those events. All of these tasks, and many of the steps in-between, generate data that the Splunk software records into log files.

Logging locations

The Splunk software internal logs are located in: $SPLUNK_HOME/var/log/splunk. This path is monitored by default, and the contents are sent to the _internal index. If the Splunk software is configured as a Forwarder, a subset of the logs are monitored and sent to the indexing tier.

The Splunk Introspection logs are located in $SPLUNK_HOME/var/log/introspection. These logs record data about the impact of the Splunk software on the host system. This path is monitored by default, and the contents are sent to the _introspection index. If the Splunk software is configured as a Forwarder, the monitored logs are sent to the indexing tier. See About Splunk Enterprise platform instrumentation.

The Splunk search logs are located in sub-folders under $SPLUNK_HOME/var/run/splunk/dispatch/. These logs record data about a search, including run time and other performance metrics. The search logs are not indexed by default. See Dispatch directory and search artifacts in the Search Manual.

Internal logs

A list of the internal logs in $SPLUNK_HOME/var/log/splunk with descriptions of their use.

Log file name	Useful for?
audit.log	Information about user activities such as a failed or successful user log in, modifying a setting, updating a lookup file, or running a search. For example, if you're looking for information about a saved search, audit.log matches the name of a saved search (savedsearch_name) with its search ID (search_id), user, and time fields. With the search_id, you can review the logs of a specific search in the search dispatch directory. See search dispatch directory in the Search Manual and audit events in the Securing Splunk Manual. Audit.log is the only log indexed to the _audit index.
btool.log	A log of btool activity. See btool.
conf.log	Contains messages about configuration replication related to Search Head Clustering. See search head clustering in the Distributed Search manual.
configuration_change.log	Contains a record of changes to Splunk Enterprise .conf files, including the creation, updating, or deletion of new .conf files in the monitored file paths. The changes are written to the log, and indexed in the _configtracker index. There are options to prevent specific .conf files, and stanzas within .conf files from being monitored. To review the default tracking behavior, see Configuration Change Tracker.
export_metrics.log	Log of Hadoop Connect metrics. See metrics related to exporting data for Hadoop Connect in the Deploy and Use Splunk Hadoop Connect manual.
first_install.log	Shows version number.
http_event_collector_metrics.log	HTTP Event Collector saves metrics about itself to this log file. See Troubleshoot HTTP Event Collector in the Getting Data In manual.
kvstore.log	Log of metrics for KV store.
license_usage.log	Indexed volume in bytes per pool, index, source, source type, and host. Available only on a Splunk instance configured as a license manager.
license_usage_summary.log	Daily indexed volume in bytes per pool, stack, and host. Available only on a Splunk instance configured as a license manager. The log in indexed into _telemetry. See Share data in Splunk Enterprise in the Admin Manual.
metrics.log	Contains periodic snapshots of Splunk performance and system data, including information about CPU usage by internal processors and queue usage in Splunk's data processing. The metrics.log file is a sampling of the top ten items in each category in 30 second intervals, based on the size of _raw. It can be used for limited analysis of volume trends for data inputs. See About metrics.log and Work with metrics.log.
migration.log	A log of events during install and migration. Specifies which files were altered during upgrade.
mongod.log	Contains runtime messages from the Splunk Enterprise KVStore. See App key value store in the Admin Manual.
python.log	Python events within Splunk. Useful for debugging REST endpoints, communication with splunkd, PDF Report Server App, Splunk Web display issues, sendmail (email alerts), and scripted or modular inputs. This log records "WARNING" instead of "WARN" for second most verbose logging level. The python.log is unmanaged by the Splunk platform. To manage the log file rotation, use an external log management service.
remote_searches.log	Messages from StreamedSearch channel. This code is executed on the search peers when a search head makes a search request. This file contains useful information on indexers regarding searches they're participating in.
scheduler.log	All actions (successful or unsuccessful) performed by the splunkd search and alert scheduler. Typically, this shows scheduled search activity.
search_messages.log	A digest of any critical messages recorded in the info.csv of all dispatched searches. The log is updated when DispatchReaper reaps the dispatch directories. Enabled by default. See limits.conf in the Admin Manual.
searches.log	No longer used. Instead, use the following search syntax: | history. This shows all the searches that have been run, plus stats for the searches.
splunkd.log	The primary log for the Splunk server. The log is often requested by Splunk Support for troubleshooting purposes. In addition, any stderr messages generated by scripted inputs, scripted search commands, and similar are logged here.
splunkd_access.log	Any action done from splunkd through the UI is logged here, including splunkweb, the CLI, all POST GET actions, deleted saved searches, and other programs accessing the REST endpoints. Also logs the time taken to respond to the requests. Search job artifacts logged here include size of data returned with search.
splunkd_stderr.log	The Unix standard error device for the server. Typically this contains (for *nix) times of healthy start and stop events, as well as various errors like exceptions, assertions, and errors generated by libraries and the operating system.
splunkd_stdout.log	The Unix standard output device for the server.
splunkd_ui_access.log	Starting in 6.2, contains a significant portion of the types of events that used to be logged in web_access.log.
splunkd-utility.log	This log is written to by the prereq-checking utils splunkd clone-prep-clear-config, splunkd validatedb, splunkd check-license, splunkd check-transforms-keys, and splunkd rest (for offline CLI). Each util logs Splunk version, some basic config, and current OS limits like max number of threads, and then messages specific to the util. Consult this log file when splunkd didn't start.
web_access.log	Requests made of Splunk Web, in an Apache access_log format. Much of the types of events logged here are logged in splunkd_ui_access.log starting in 6.2.
web_service.log	Primary log written by splunkweb. Records actions made by splunkweb. Note: the log records "WARNING" instead of "WARN" for second most verbose logging level.

Some log files are not created until your Splunk instance uses them. Other logs are created, but will remain empty until events are written.

The log management process

The internal logs are rolled based on file size, with a number of historical logs kept. The historical rotation for most internal logs is 5 files of 25MB each. You can review the log rotation settings in $SPLUNK_HOME/etc/log.cfg.

For long-term changes to the log management process, such as increasing the historical log rotation or log size, we recommend creating a $SPLUNK_HOME/etc/log-local.cfg file and placing your changes in there. The settings in log-local.cfg take precedence over log.cfg, and the file does not get overwritten on upgrade.

Logging levels

Splunk platform internal logging levels are DEBUG INFO WARN ERROR FATAL from most to least verbose. The debug logging is disabled by default. See enabling debug logging.

Use Splunk Web to manage logging-level

To change the logging-level using Splunk Web:

1. Navigate to Settings > Server settings > Server logging. This generates a list of log channels and their status.

2. To change the logging level for a particular log channel, click on that channel. This brings up a page specific to that channel.

3. On the log channel's page, change the logging level.

When you change the logging level, note the following:

• The change is immediate and dynamic.
• The change is not persistent; it goes away when the Splunk service is restarted.

Searching internal logs

By default, only the Admin role can search the _internal index, and it must be defined in the search explicitly. Search the internal log files in Splunk Web by typing:

index=_internal

Search for errors and warnings by typing:

index=_internal (log_level=error OR log_level=warn*)

Search the internal logs using Pivot

Splunk Enterprise includes data models constructed from the internal logs. To access the internal log data models, in the Search & Reporting app in Splunk Web, click Datasets.

Set logging levels and log channels for a search

You can use the noop command to set the logging level and logging channel for a specific search job. Use the log_<level> argument to identify a logging level and one or more logging channels. For more information see noop in the Search Reference Manual.

Configuration Change Tracker

Splunk Enterprise configuration settings are stored in configuration files. These files are identified by the .conf extension. Changes to the .conf files are made using Splunk Web, a Splunk platform configuration management node such as the cluster manager or deployment server, a 3rd party configuration management tool, or by editing the .conf files directly. To maintain a record of the changes for troubleshooting and maintenance purposes, Splunk Enterprise monitors the .conf files, and logs the changes made to them by default.

When Splunk Enterprise services are running, and a change is made to the .conf files, a checksum is calculated and logged with the full .conf file path. The contents of the log can be viewed in the _configtracker index by a user in the Admin role. The indexing of this data does not count towards your license.

By default, only the Admin role can search the _configtracker index, and the index name must be defined in the search explicitly.

Events in the _configtracker index contain a record of changes to Splunk Enterprise .conf files, including the creation, updating, or deletion of new .conf files in the monitored file paths. Specifically, config change tracking includes only changes that might have an effect your environment. For example, if a file with a stanza and key-value pair is created, updated, or deleted, the change is logged. However, if an empty file or a stanza with no key-value pairs is added or deleted, the change is not logged since it has no effect on the environment. Similarly, comments added to or deleted from files are not tracked.

The monitored paths and files are:

File path
$SPLUNK_HOME/etc/system
$SPLUNK_HOME/etc/apps
$SPLUNK_HOME/etc/users
$SPLUNK_HOME/etc/peer-apps
$SPLUNK_HOME/etc/instance.cfg

The file monitoring is limited to the default and local folders under the specified paths.

If a valid .conf file in one of these file paths is changed while Splunk software is stopped, the change is tracked and logged when the software restarts. For a list of valid .conf files, see List of configuration files and their context.

Change the configuration tracking behavior

There are two methods used to check for .conf file changes: monitoring and polling. The monitor uses the inotify API available in some Linux distributions. If inotify isn't detected, polling is used. Polling checks for changes to the file paths and files every 30 seconds.

When polling is in use, a change to a .conf file can take 30 seconds to appear in the log.

You can use the throttling option to reduce the tracking messaging, and summarize any changes to the .conf files in one event. To use throttling, you must be running Splunk Enterprise on Linux with inotify detected.

To review the configuration tracking options, see the config_change_tracker stanza in server.conf.

Masking fields in configuration files

To prevent hashed passwords and other sensitive data from being recorded and indexed, the configuration tracking ignores the values in some fields by default. A change to the field value is recorded only as a change; no content is logged.

Field names with ignored values
pass4SymmKey, sslPassword, bindDNpassword, password, token, integrationKey, secretKey, appSecretKey, accessKey, scriptSecureArguments, socksPassword, auth_password, attributeQuerySoapPassword, remote.azure.access_key, remote.azure.secret_key, remote.s3.access_key, remote.s3.secret_key, remote.s3.kms.key_id

If there are additional fields you want to mask in a .conf file, use the exclude_fields setting in server.conf to define the file, stanza, and key field to ignore. When using exclude_fields, a change to a matching field is not logged.

For example, to prevent the logging of the username value in identities.conf, configure the exclude_fields setting in the format file:stanza:key.

[config_change_tracker]
exclude_fields = identities.conf:mysql_local:username

For more information, see the config_change_tracker stanza in server.conf.

Remove configuration files from being monitored

If you want to remove an entire .conf file from being monitored for changes, use the denylist setting under the [config_change_tracker] stanza. The denylist accepts a regular expression. To learn more about Splunk Enterprise and the use of regular expressions, see About Splunk regular expressions in the Knowledge Manager Manual.

Example	*nix	Windows
Ignore all .conf files in the search app	denylist=/etc/apps/search	denylist=\\etc\\apps\\search
Ignore a savedsearches.conf file in both the default and local folders of the search app	denylist=/etc/apps/search/(?:default|local)/savedsearches\.conf$	denylist=\\etc\\apps\\search\\(?:default|local)\\savedsearches\.conf$