Admin Manual

About configuration files

Splunk's configuration information is stored in configuration files, identified by their .conf extension. These files are located under $SPLUNK_HOME/etc.

When you make a change to a configuration setting in Splunk Manager in Splunk Web, the change gets written to the relevant configuration file. This change is written to a copy of the configuration file in a directory under $SPLUNK_HOME/etc (the actual directory depends on a number of factors, discussed later), and the default value of the attribute is left alone in $SPLUNK_HOME/etc/system/default.

You can do a lot of configuration from Manager, but for some more advanced customizations, you must edit the configuration files directly.

For a complete list of configuration files, see "List of configuration files".

The configuration directory structure

The following is the configuration directory structure that exists under $SPLUNK_HOME/etc:

• $SPLUNK_HOME/etc/system/default
  • This contains the pre-configured configuration files. Do not modify the files in this directory.
• $SPLUNK_HOME/etc/system/local
  • Local changes on a site-wide basis go here; for example, settings you want to make available to all apps. If the configuration file you're looking for doesn't already exist in this directory, create it and give it write permissions.
• $SPLUNK_HOME/etc/slave-apps/[_cluster|<app_name>]/[local|default]
  • For cluster peer nodes only.
  • The subdirectories under $SPLUNK_HOME/etc/slave-apps contain configuration files that are common across all peer nodes.
  • Do not change the content of these subdirectories on the cluster peer itself. Instead, use the cluster master to distribute any new or modified files to them.
  • The _cluster directory contains configuration files that are not part of real apps but that still need to be identical across all peers. A typical example is the indexes.conf file.
  • For more information, see "Update common peer configurations" in the Managing Indexers and Clusters manual.
• $SPLUNK_HOME/etc/apps/<app_name>/[local|default]
  • If you're in an app when a configuration change is made, the setting goes into a configuration file in the app's /local directory. For example, edits for search-time settings in the default Splunk search app go here: $SPLUNK_HOME/etc/apps/search/local/.
  • If you want to edit a configuration file so that the change only applies to a certain app, copy the file to the app's /local directory (with write permissions) and make your changes there.
• $SPLUNK_HOME/etc/users
  • User-specific configuration changes go here.
• $SPLUNK_HOME/etc/system/README
  • This directory contains supporting reference documentation. For most configuration files, there are two reference files: .spec and .example; for example, inputs.conf.spec and inputs.conf.example. The .spec file specifies the syntax, including a list of available attributes and variables. The .example file contains examples of real-world usage.

A single Splunk instance typically has multiple versions of some configuration files, across several of these directories. For example, you can have configuration files with the same names in your default, local, and app directories. This provides a layering effect that allows Splunk to determine configuration priorities based on factors such as the current user and the current app. Be sure to review the topic "Configuration file precedence" to understand the precedence rules governing Splunk configuration files. That topic explains how Splunk determines which files have priority.

Note: The most accurate list of settings available for a given configuration file is in the .spec file for that configuration file. You can find the latest version of the .spec and .example files in the "Configuration file reference", or in $SPLUNK_HOME/etc/system/README.

The default directory

"all these worlds are yours, except /default - attempt no editing there"

-- duckfez, 2010

Default configuration files are located in $SPLUNK_HOME/etc/system/default. Never edit the default version of a configuration file. Default files get overwritten each time you upgrade Splunk.

Instead of touching the default files, create or edit versions of the files in any of the other configuration directories, such as $SPLUNK_HOME/etc/system/local. These directories do not get overwritten during upgrades.

Splunk always looks at the default directory last, so any attributes or stanzas that you change in one of the other configuration directories will take precedence over the default version. You can layer several versions of a configuration file on top of one-another, with different attribute values filtering through and being used by Splunk, according to the layering scheme described in "Configuration file precedence". For most deployments, however, you can just use the $SPLUNK_HOME/etc/system/local directory to make configuration changes.

It is also a bad idea to copy a default configuration file to another configuration directory and then edit the copy there. If a default configuration file gets changed by a Splunk upgrade, the change might get blocked by the version of the file you've copied. For example, consider a scenario where some default configuration file has some attribute with a value of 50. Then the Splunk engineering organization subsequently determines that the attribute works better if set to 100, rather than 50. So they change the attribute's value to 100 in a subsequent release. When you next upgrade Splunk, the new version of the file, with the attribute value of 100, overwrites the old version. That's all well and good. However, if in the meantime you've put a copy of the previous version of the file in $SPLUNK_HOME/etc/system/local, Splunk will continue to use the value of 50, since attribute values in $SPLUNK_HOME/etc/system/local take precedence over values in the default directory.

Under limited circumstances and with great care, you can, however, copy a stanza from a default file, edit it, and then put the edited stanza in a new version of the file in $SPLUNK_HOME/etc/system/local or some other such location. But only do this for stanzas and attributes that you need to edit; for the reasons described above, do not copy over more of the configuration file than you actually need to change.

Note: Some configuration files do not have default versions. These configuration files still have .spec and .example files you can look at.

Creating and editing configuration files on non-UTF-8 operating systems

Splunk expects configuration files to be in ASCII/UTF-8. If you are editing or creating a configuration file on an operating system that is non-UTF-8, you must ensure that the editor you are using is configured to save in ASCII/UTF-8.

The structure of configuration files

Configuration files consist of one or more stanzas, or sections. Each stanza begins with a stanza header, designated by square brackets. Following the header is a series of attribute/value pairs that specify configuration settings. Depending on the stanza type, some of the attributes might be required, while others could be optional.

Here's the basic pattern:

[stanza1_header]
<attribute1> = <val1>
# comment 
<attribute2> = <val2>
...

[stanza2_header]
<attribute1> = <val1>
<attribute2> = <val2>
...

Important: Attributes are case-sensitive. For example, sourcetype = my_app is not the same as SOURCETYPE = my_app. One will work; the other won't.

Stanza scope

Configuration files frequently have stanzas with varying scopes, with the more specific stanzas taking precedence. For example, consider this example of an outputs.conf configuration file, used to configure forwarders:

[tcpout]
indexAndForward=true
compressed=true

[tcpout:my_indexersA]
autoLB=true
compressed=false
server=mysplunk_indexer1:9997, mysplunk_indexer2:9997

[tcpout:my_indexersB]
autoLB=true
server=mysplunk_indexer3:9997, mysplunk_indexer4:9997

This example file has two levels of stanzas:

• The global [tcpout], with settings that affect all tcp forwarding.
• Two [tcpout:<target_list>] stanzas, whose settings affect only the indexers defined in each target group.

The setting for compressed in [tcpout:my_indexersA] overrides that attribute's setting in [tcpout], for the indexers in the my_indexersA target group only.

For more information on forwarders and outputs.conf, see Configure forwarders with outputs.conf.

A few syntax notes

There are a couple of miscellaneous things to keep in mind when editing configuration files.

Clear attributes

You can clear any attribute by setting it to null. For example:

forwardedindex.0.whitelist = 

This overrides any previous value that the attribute held, including any value set in its default file, causing the system to consider the value entirely unset.

Use comments

You can insert comments in configuration files. To do so, use the # sign:

# This stanza forwards some log files.
[monitor:///var/log]

Important: Start the comment at the left margin. Do not put the comment on the same line as the stanza or attribute:

[monitor:///var/log]    # This is a really bad place to put your comment.

For an attribute, such as

a_setting = 5  #5 is the best number

This sets the a_setting attribute to the value "5 #5 is the best number", which may cause unexpected results.