Configuration file precedence
Splunk software uses configuration files to determine nearly every aspect of its behavior. A Splunk platform deployment can have many copies of the same configuration file. These file copies are usually layered in directories that affect either the users, an app, or the system as a whole.
When editing configuration files, it is important to understand how Splunk software evaluates these files and which ones take precedence.
When incorporating changes, Splunk software does the following to your configuration files:
- It merges the settings from all copies of the file, using a location-based prioritization scheme.
- When different copies have conflicting attribute values (that is, when they set the same attribute to different values), it uses the value from the file with the highest priority.
- It determines the priority of configuration files by their location in the directory structure, according to the rules described in this topic.
Note: Besides resolving configuration settings among multiple copies of a file, Splunk software sometimes needs to resolve settings within a single file. See Attribute precedence within a single props.conf file.
About configuration file context
To determine the order of directories for evaluating configuration file precedence, Splunk software considers each file's context. Configuration files operate in either a global context or in the context of the current app and user:
- Global. Activities like indexing take place in a global context. They are independent of any app or user. For example, configuration files that determine monitoring or indexing behavior occur outside of the app and user context and are global in nature.
- App/user. Some activities, like searching, take place in an app or user context. The app and user context is vital to search-time processing, where certain knowledge objects or actions might be valid only for specific users in specific apps.
The precedence order for configuration file directories varies according to the context of the particular configuration file. To learn the context of each file, see List of configuration files and their context.
How Splunk determines precedence order
Configuration file precedence order depends on the location of file copies within the directory structure. Splunk software considers the context of each file to determine the precedence order of the directories.
Precedence within global context
When the file context is global, directory priority descends in this order:
1. System local directory -- highest priority
2. App local directories
3. App default directories
4. System default directory -- lowest priority
When consuming a global configuration, such as
inputs.conf, Splunk software first uses the attributes from any copy of the file in
system/local. Then it looks for any copies of the file located in the app directories, adding any attributes found in them, but ignoring attributes already discovered in system/local. As a last resort, for any attributes not explicitly assigned at either the system or app level, it assigns default values from the file in the
Note: As the next section describes, cluster peer nodes have an expanded order of precedence.
Precedence within global context, indexer cluster peers only
There is an expanded precedence order for indexer cluster peer configurations, which are considered in the global context. This is because some configuration files, like
indexes.conf, must be identical across peer nodes.
To keep configuration settings consistent across peer nodes, configuration files are managed from the cluster manager node, which pushes the files to the
slave-app directories on the peer nodes. Files in the
slave-app directories have the highest precedence in a cluster peer's configuration. These directories exist only on indexer cluster peer nodes.
Here is the expanded precedence order for cluster peers:
1. Slave-app local directories -- highest priority
2. System local directory
3. App local directories
4. Slave-app default directories
5. App default directories
6. System default directory -- lowest priority
Precedence within app or user context
For files with an app/user context, directory priority descends from user to app to system:
1. User directories for current user -- highest priority
2. App directories for currently running app (local, followed by default)
3. App directories for all other apps (local, followed by default) -- for exported settings only
4. System directories (local, followed by default) -- lowest priority
An attribute in
savedsearches.conf, for example, might be set at all three levels: the user, the app, and the system. Splunk will always use the value of the user-level attribute, if any, in preference to a value for that same attribute set at the app or system level.
How app directory names affect precedence
For most practical purposes, the information in this subsection probably won't matter, but it might prove useful if you need to force a certain order of evaluation or for troubleshooting.
The effect of app directory names varies depending on whether the context is global or local.
App directory names in the global context
When determining priority in the global context, Splunk software uses lexicographical order to determine priority among the collection of apps directories. For example, files in an apps directory named "A" have a higher priority than files in an apps directory named "B", and so on.
App directory names in the app/user context
When determining priority in the app/user context, Splunk software uses reverse-lexicographical order to determine priority among the collection of apps directories, For example, files in an apps directory named "B" have a higher priority than files in an apps directory named "A", and so on.
When determining precedence in the app/user context, directories for the currently running app take priority over those for all other apps, independent of how they're named. Furthermore, other apps are only examined for exported settings.
The finer points of lexicographical order
In the global context only, lexicographical order determines precedence. Thus, files in an apps directory named "A" have a higher priority than files in an apps directory named "B", and so on. Also, all apps starting with an uppercase letter have precedence over any apps starting with a lowercase letter, due to lexicographical order. ("A" has precedence over "Z", but "Z" has precedence over "a", for example.)
In addition, numbered directories have a higher priority than alphabetical directories and are evaluated in lexicographic, not numerical, order. For example, in descending order of precedence:
$SPLUNK_HOME/etc/apps/myapp1 $SPLUNK_HOME/etc/apps/myapp10 $SPLUNK_HOME/etc/apps/myapp2 $SPLUNK_HOME/etc/apps/myapp20 ... $SPLUNK_HOME/etc/apps/myappApple $SPLUNK_HOME/etc/apps/myappBanana $SPLUNK_HOME/etc/apps/myappZabaglione ... $SPLUNK_HOME/etc/apps/myappapple $SPLUNK_HOME/etc/apps/myappbanana $SPLUNK_HOME/etc/apps/myappzabaglione ...
Lexicographical order sorts items based on the values used to encode the items in computer memory. In Splunk software, this is almost always UTF-8 encoding, which is a superset of ASCII.
- Numbers are sorted before letters. Numbers are sorted based on the first digit. For example, the numbers 10, 9, 70, 100 are sorted lexicographically as 10, 100, 70, 9.
- Uppercase letters are sorted before lowercase letters.
- Symbols are not standard. Some symbols are sorted before numeric values. Other symbols are sorted before or after letters.
In the app/user context, precedence is determined instead by reverse-lexicographical order. Therefore, the order of precedence is exactly opposite the lexicographical order described above, which is used in the global context only. For example, files in an apps directory named "B" have a higher priority than files in an apps directory named "A", files in app "a" have precedence over files in apps "B" or "A", and so on. Similarly, numerical app directories have a lower precedence than alphabetical directories.
Summary of the effect of directories on configuration precedence
Putting this all together, the order of directory priority, from highest to lowest, goes like this:
$SPLUNK_HOME/etc/system/local/* $SPLUNK_HOME/etc/apps/A/local/* ... $SPLUNK_HOME/etc/apps/z/local/* $SPLUNK_HOME/etc/apps/A/default/* ... $SPLUNK_HOME/etc/apps/z/default/* $SPLUNK_HOME/etc/system/default/*
Global context, cluster peer nodes only
$SPLUNK_HOME/etc/slave-apps/A/local/* ... $SPLUNK_HOME/etc/slave-apps/z/local/* $SPLUNK_HOME/etc/system/local/* $SPLUNK_HOME/etc/apps/A/local/* ... $SPLUNK_HOME/etc/apps/z/local/* $SPLUNK_HOME/etc/slave-apps/A/default/* ... $SPLUNK_HOME/etc/slave-apps/z/default/* $SPLUNK_HOME/etc/apps/A/default/* ... $SPLUNK_HOME/etc/apps/z/default/* $SPLUNK_HOME/etc/system/default/*
slave-apps/[local|default] directories, the special
_cluster subdirectory has a higher precedence than any app subdirectories starting with a lowercase letter (for example,
anApp). However, it has a lower precedence than any apps starting with an uppercase letter (for example,
AnApp). This is due to the location of the underscore ("_") character in the lexicographical order.
$SPLUNK_HOME/etc/users/* $SPLUNK_HOME/etc/apps/Current_running_app/local/* $SPLUNK_HOME/etc/apps/Current_running_app/default/* $SPLUNK_HOME/etc/apps/z/local/*, $SPLUNK_HOME/etc/apps/z/default/*, ... $SPLUNK_HOME/etc/apps/A/local/*, $SPLUNK_HOME/etc/apps/A/default/* $SPLUNK_HOME/etc/system/local/* $SPLUNK_HOME/etc/system/default/*
In the app/user context, all configuration files for the currently running app take priority over files from all other apps. This is true for both the app's local and default directories. So, if the current context is app C, Splunk evaluates both
$SPLUNK_HOME/etc/apps/C/default/* before evaluating the local and default directories for any other apps. Furthermore, Splunk software only looks at configuration data for other apps if that data has been exported globally through the app's
Also note that
/etc/users/ is evaluated only when the particular user logs in or performs a search.
Example of how attribute precedence works
This example of attribute precedence uses
props.conf file is unusual, because its context can be either global or app/user, depending on when Splunk is evaluating it. Splunk evaluates
props.conf at both index time (global) and search time (apps/user).
$SPLUNK_HOME/etc/system/local/props.conf contains this stanza:
[source::/opt/Locke/Logs/error*] sourcetype = fatal-error
$SPLUNK_HOME/etc/apps/t2rss/local/props.conf contains another version of the same stanza:
[source::/opt/Locke/Logs/error*] sourcetype = t2rss-error SHOULD_LINEMERGE = True BREAK_ONLY_BEFORE_DATE = True
The line merging attribute assignments in
t2rss always apply, as they only occur in that version of the file. However, there's a conflict with the
sourcetype attribute. In the
/system/local version, the
sourcetype has a value of "fatal-error". In the
/apps/t2rss/local version, it has a value of "t2rss-error".
Since this is a
sourcetype assignment, which gets applied at index time, Splunk uses the global context for determining directory precedence. In the global context, Splunk gives highest priority to attribute assignments in
system/local. Thus, the
sourcetype attribute gets assigned a value of "fatal-error".
The final, internally merged version of the file looks like this:
[source::/opt/Locke/Logs/error*] sourcetype = fatal-error SHOULD_LINEMERGE = True BREAK_ONLY_BEFORE_DATE = True
List of configuration files and their context
As mentioned, Splunk decides how to evaluate a configuration file based on the context that the file operates within, global or app/user. Generally speaking, files that affect data input, indexing, or deployment activities are global; files that affect search activities usually have a app/user context.
transforms.conf files can be evaluated in either a app/user or a global context, depending on whether Splunk is using them at index or search time. The
limits.conf file is evaluated in a global context except for a few settings, which are tunable by app or user.
Global configuration files
admon.conf authentication.conf authorize.conf crawl.conf deploymentclient.conf distsearch.conf indexes.conf inputs.conf limits.conf, except for indexed_realtime_use_by_default outputs.conf pdf_server.conf procmonfilters.conf props.conf -- global and app/user context pubsub.conf regmonfilters.conf report_server.conf restmap.conf searchbnf.conf segmenters.conf server.conf serverclass.conf serverclass.seed.xml.conf source-classifier.conf sourcetypes.conf sysmon.conf tenants.conf transforms.conf -- global and app/user context user-seed.conf -- special case: Must be located in /system/default web.conf wmi.conf
App/user configuration files
alert_actions.conf app.conf audit.conf commands.conf eventdiscoverer.conf event_renderers.conf eventtypes.conf fields.conf literals.conf macros.conf multikv.conf props.conf -- global and app/user context savedsearches.conf tags.conf times.conf transactiontypes.conf transforms.conf -- global and app/user context user-prefs.conf workflow_actions.conf
Troubleshooting configuration precedence and other issues
Splunk's configuration file system supports many overlapping configuration files in many different locations. The price of this level of flexibility is that figuring out which value for which configuration option is being used in your Splunk installation can sometimes be quite complex. If you're looking for some tips on figuring out what configuration setting is being used in a given situation, read Use btool to troubleshoot configurations in the Troubleshooting Manual.
Configuration file structure
Attribute precedence within a single props.conf file
This documentation applies to the following versions of Splunk® Enterprise: 8.1.0, 8.1.1, 8.1.2, 8.1.3, 8.1.4, 8.1.5, 8.1.6, 8.1.7, 8.1.8, 8.1.9, 8.1.10, 8.2.0, 8.2.1, 8.2.2, 8.2.3, 8.2.4, 8.2.5, 8.2.6