Splunk® Enterprise

Troubleshooting Manual

Download manual as PDF

Splunk Enterprise version 5.0 reached its End of Life on December 1, 2017. Please see the migration information.
This documentation does not apply to the most recent version of Splunk. Click here for the latest version.
Download topic as PDF

What Splunk logs about itself

Splunk keeps track of its activity by logging to various files in $SPLUNK_HOME/var/log/splunk.

Splunk's internal log files are rolled based on size. You can change the default log rotation size by editing $SPLUNK_HOME/etc/log.cfg.

Search these files in Splunk Web by typing:

index=_internal

Internal logs

Here is a list, with descriptions, of the internal logs in $SPLUNK_HOME/var/log/splunk. Splunk's internal logs are useful for troubleshooting or metric analysis.

Note that some log files are not created until your Splunk instance uses them, for example crawl.log.

Log file name Useful for?
audit.log Stats about user activity. For example, if you're looking for information about a saved search, audit.log matches the name of your saved search (savedsearch_name) with its search ID (search_id), user, and time. With the search_id, you can look up that particular search elsewhere, like in the search dispatch directory. Read about audit events in the Securing Splunk Manual.
btool.log Log of btool activity. Read about btool in this manual.
crawl.log Log of crawl activity. Read about crawl in the Getting Data In Manual. Crawl is now deprecated.
first_install.log Shows version number.
inputs.log Inputs found by the deprecated command crawl. This log file will be empty unless you use the crawl command.
intentions.log Intentions activity. Read about intentions in the Developing Views and Apps for Splunk Web Manual.
license_audit.log Continuous audit of license violations. If you have a license master and slaves, look at license_usage.log instead of here. (As soon as a license master's features such as pooling or slaves are configured, license_audit.log stops logging events.)
license_usage.log Indexed volume in bytes per pool, source, sourcetype, and host. Starting in 4.2, license_usage.log is available only on a Splunk license master.
metrics.log Contains periodic snapshots of Splunk performance and system data, including information about CPU usage by internal processors and queue usage in Splunk's data processing. The metrics.log file is a sampling of the top ten items in each category in 30 second intervals, based on the size of _raw. It can be used for limited analysis of volume trends for data inputs. For more information about metrics.log, see About metrics.log and Work with metrics.log in this manual.
migration.log A log of events during install and migration. Specifies which files were altered during upgrade.
python.log Python events within Splunk. Useful for debugging REST endpoints, communication with splunkd, PDF Report Server App, Splunk Web display issues, sendmail (email alerts), and scripted inputs. With web_service.log, one of the few Splunk logs that uses "WARNING" instead of "WARN" for second most verbose logging level.
scheduler.log All actions (successful or unsuccessful) performed by the splunkd search and alert scheduler. Typically, this shows scheduled search activity.
searches.log Beginning with Splunk 5, no longer used. Instead, use the following search syntax: | history. This shows all the searches that have been run, plus stats for the searches.
searchhistory.log No longer used.
splunkd.log The primary log written to by the Splunk server. May be requested by Splunk Support for troubleshooting purposes. Any stderr messages generated by scripted inputs, scripted search commands, and so on, are logged here.
splunkd_access.log Any action done from splunkd through the UI is logged here, including splunkweb, the CLI, all POST GET actions, deleted saved searches, and other programs accessing the REST endpoints. Also logs the time taken to respond to the requests. sourcetype="splunkd_access.log"
splunkd_stderr.log The Unix standard error device for the server. Typically this contains (for *nix) times of healthy start and stop events, as well as various errors like exceptions, assertions, and errors generated by libraries and the operating system.
splunkd_stdout.log The Unix standard output device for the server.
web_access.log Requests made of Splunk Web, in an Apache access_log format.
web_service.log Primary log written by splunkweb. Records actions made by splunkweb. This and python.log are the only logs that, in second most verbose logging level, write messages with "WARNING" instead of Splunk log files' usual "WARN."

Splunk search logs

Splunk also creates search logs. Note that these are not indexed to _internal.

Each search has its own directory for all information specific to the search, including its search logs. The search's directory is named with (among other parameters) the search_id. (Match a search to its search_id in audit.log.) You'll find the search directory in $SPLUNK_HOME/var/run/splunk/dispatch/.

If you have any long-running real-time searches, you might want to adjust the maximum size of your search logs. These logs are rotated when they reach a default maximum size of 10 MB. Splunk keeps up to five of them for each search, so the total log size for a search can conceivably grow as large as 30 MB.

Most searches are unlikely to generate logs anywhere near 10 MB in size; however, it can become an issue if you have ongoing real-time searches.

To adjust the log size, edit $SPLUNK_HOME/etc/log-searchprocess.cfg.

Debug mode

Splunk has a debugging parameter. Read about enabling debug logging in this manual.

Except where noted above, Splunk's internal logging levels are DEBUG INFO WARN ERROR FATAL (from most to least verbose).

Note: Running Splunk with debugging turned on outputs a large amount of information. Make sure you do not leave debugging on for any significant length of time.

Use Splunk Web to manage logs

To view and manage logs, you can use the Manager:

1. Navigate to Manager > System settings > System logging. This generates a list of log channels and their status.

2. To change the logging level for a particular log channel, click on that channel. This brings up a page specific to that channel.

3. On the log channel's page, you can change its logging level.

When you change the logging level, note the following:

  • The change is immediate and dynamic.
  • The change is not persistent; it goes away when Splunk is restarted.

Manager > System settings > System logging is meant only for dynamic and temporary changes to Splunk log files. For permanent changes, use $SPLUNK_HOME/etc/log.cfg instead.

PREVIOUS
Splunk on Splunk app
  NEXT
Enable debug logging

This documentation applies to the following versions of Splunk® Enterprise: 5.0, 5.0.1, 5.0.2, 5.0.3, 5.0.4, 5.0.5, 5.0.6, 5.0.7, 5.0.8, 5.0.9, 5.0.10, 5.0.11, 5.0.12, 5.0.13, 5.0.14, 5.0.15, 5.0.16, 5.0.17, 5.0.18


Was this documentation topic helpful?

Enter your email address, and someone from the documentation team will respond to you:

Please provide your comments here. Ask a question or make a suggestion.

You must be logged into splunk.com in order to post comments. Log in now.

Please try to keep this discussion focused on the content covered in this documentation topic. If you have a more general question about Splunk functionality or are experiencing a difficulty with Splunk, consider posting a question to Splunkbase Answers.

0 out of 1000 Characters