How do configuration files work?

How do configuration files work?

Splunk's configurations are affected via configuration files. Even configurations set up through Splunk Web or the CLI are written out to configuration files. Set up more advanced configurations in configuration files, or make an application. Learn more about application configuration, including best practices.

Once you have created a working application for a single Splunk server, you can then distribute it to target servers through the Splunk deployment server or share them with others through SplunkBase.

Changes to how Splunk processes index data do not affect data that is already indexed.

Configuration files must be created in ASCII or UTF8 character sets.

Restarting after configuration changes

Many configuration file changes require you to restart Splunk. Check the configuration file and/or its documentation reference topic to see if a change you make requires you to restart Splunk.

The following changes require additional or different actions before they will take effect:

• Enable search-time configuration changes made to transforms.conf and props.conf by typing the following search in Splunk Web:

• Bounce authentication.conf via the Admin -> Server section of Splunk Web.

Configuration file directory structure

There are two general configuration file directories in $SPLUNK_HOME/etc/:

• $SPLUNK_HOME/etc/system/
  • The system directory contains default shipped Splunk configurations and user-created content to override these settings.
• $SPLUNK_HOME/etc/apps/
  • The apps directory stores downloaded and custom built applications.
  • Store custom configurations in custom directories underneath ../apps/.

NOTE: There is also a legacy directory in $SPLUNK_HOME/etc/bundles to support prior versions' configurations and the deployment server.

Both system/ and the application directories in apps/ have the same directory structure:

• default/
  • Settings in default/ should not be changed.
• local/
  • Make all custom edits here, including overriding settings in default/.
• readme/
  • Supporting documentation.
• bin/
  • Scripts that support the application, such as searchscripts, custom web scripts, and REST endpoint handlers.
• static/
  • Files served by the HTTP server and other static files (non-executable).

For example:

    apps/
        myapp1/
            default/
            local/
            static/
            bin/
        myapp2/
            default/
            local/
            static/
            bin/

Your Splunk server ships with several such directories, including:

• default - contains the pre-configured configuration files. Do not modify the files in default.

Note: Not all configuration files appear in default/.

• local - stores modifications you make through the web interface or command line. You can make file edits here, or in a custom application directory.

Note: If you edit files that are also written to by Splunk Web, your edits may be overridden if someone else is editing Splunk Web at the same time.

• learned - this set of configurations are settings created by the Splunk Server as it trains on incoming data.
• README - this directory contains example and spec configuration files that can help you create your own configuration files. For each configuration file, there are two reference files; .spec and .example. For example, inputs.conf.spec and inputs.conf.example. The .spec file is a specification of syntax, including which attributes and variables are available. The .example files are helpful examples of real-world usage. These files are all found in the $SPLUNK_HOME/etc/system/README directory.

Configuration file precedence

Configuration files live in multiple places: default, local and any custom application directories you create. Configuration files are evaluated in the following order:

• local -- local changes and preferences are evaluated first.
• user-created directories -- these are evaluated in alphabetical order.
• default -- Splunk's default settings are evaluated last.

NOTE: Any configurations set in $SPLUNK_HOME/etc/bundles take precedence over configurations in $SPLUNK_HOME/etc/apps.

Example

Directories are evaluated in the following order:

$SPLUNK_HOME/etc/system/local/*
$SPLUNK_HOME/etc/bundles/local/*
$SPLUNK_HOME/etc/bundles/A/*
...
$SPLUNK_HOME/etc/bundles/Z/* $SPLUNK_HOME/etc/system/local/*
$SPLUNK_HOME/etc/apps/A/local/*
...
$SPLUNK_HOME/etc/apps/Z/local/*
$SPLUNK_HOME/etc/apps/A/default/*
...
$SPLUNK_HOME/etc/apps/Z/default/*
$SPLUNK_HOME/etc/system/default/*

Numbered directories are evaluated in the following order:

$SPLUNK_HOME/etc/apps/myapp1
$SPLUNK_HOME/etc/apps/myapp10
$SPLUNK_HOME/etc/apps/myapp2
$SPLUNK_HOME/etc/apps/myapp20
...

Attribute precedence

Precedence is applied attribute-by-attribute. That is, if the file props.conf exists in local and a user created configuration file directory, the props.conf file in local does not override or replace the entire props.conf file. If the same attribute/specification exists in both the local props.conf and the user-created props.conf, the local props.conf overrides the attribute.

For example, if $SPLUNK_HOME/etc/system/local/props.conf contains this stanza:

[source::/opt/Locke/Logs/error*]
sourcetype = t2rss-error

And $SPLUNK_HOME/etc/apps/t2rss/props.conf contains this stanza:

[source::/opt/Locke/Logs/error*]
SHOULD_LINEMERGE = True
BREAK_ONLY_BEFORE_DATE = True

Both the sourcetype assignment in local and the line merging attributes in t2rss apply. However, if both local and t2rss have a sourcetype assignment for source::/opt/Locke/Logs/error*, the assignment in local overrides t2rss.

Precedence rules for events with multiple attribute assignments

Beyond the above rules for precedence, there is an additional precedence issue that affects only props.conf. props.conf sets attributes for processing individual events by host, source or sourcetype (and sometimes eventtype). So it's possible for one event to have the same attribute set differently for the default fields: host, source or sourcetype. The precedence order is:

• source
• host
• sourcetype

Settings higher in the list will override settings lower in the list for settings with the same name.

Note: only one stanza of each type will apply to a given event, one source, one host, and one sourcetype stanza. Therefore, you should not create overlapping source expressions, in order to achieve a defined result.

You may want to override default props.conf settings. For example, you are tailing mylogfile.xml, which by default is labeled sourcetype = xml_file. This configuration will re-index the entire file whenever it changes, even if you manually specify another sourcetype, because the property is set by source. To override this, add the explicit configuration by source:

[source::/var/log/mylogfile.xml]
CHECK_METHOD = endpoint_md5

• Was this topic useful?
• Post a comment