Admin Manual

 


Configuration file precedence

NOTE - Splunk version 4.x reached its End of Life on October 1, 2013. Please see the migration information.

This documentation does not apply to the most recent version of Splunk. Click here for the latest version.

Configuration file precedence

This topic describes how Splunk sifts through its layers of configuration files to determine which settings to use. For general information about configuration files, read "About configuration files".

Order of precedence

Splunk uses configuration files to determine nearly every aspect of its behavior. Besides having many different types of configuration files, a single Splunk instance can also have many copies of the same configuration file, layered in directories categorized by user, app, and system. When consuming a configuration file, Splunk merges the settings from all copies of the file, using a location-based prioritization scheme. When different copies have conflicting attribute values (that is, when they set the same attribute to different values), Splunk uses the value from the file with the highest priority.

Splunk determines the priority of configuration files by their location in its directory structure. At the most general level, it prioritizes according to whether the file is located in a system, app, or user directory.

You need to understand how this works so that you can manage your Splunk configurations intelligently. It's all pretty straightforward when you focus on context. Once you get a feel for the context in which Splunk is consuming a particular file, the way precedence works makes quite a bit of sense.

Note: Besides resolving configuration settings amongst multiple copies of a file, Splunk sometimes needs to resolve settings within a single file. For information on how Splunk determines precedence within a single props.conf file, see "Attribute precedence within a single props.conf file".

The app/user context

In determining priority among copies of a configuration file, Splunk uses two different schemes of directory precedence, according to whether that particular configuration relates to an activity with an app/user context -- that is, where the current app and user matter. Some activities, like searching, take place in an app/user context; others, like indexing, take place in a global context, independent of any app or user.

For instance, configuration files that determine indexing or monitoring behavior occur outside of the app/user context; they are global in nature. At the time of data input or event indexing, it does not matter which user or app might later access the data or event. The app/user context, on the other hand, is vital to search-time processing, where certain knowledge objects or actions might be valid only for specific users in specific apps.

How context affects precedence order

When the context is global, where there's no app/user context, directory priority descends from system/local to app to system/default:

  • System local directory -- highest priority.
  • App directories (local overrides default).
  • System default directory -- lowest priority.

When consuming a global configuration, such as inputs.conf, Splunk first gets the attributes from any copy of the file in system/local. Then it looks for any copies of the file located in the app directories, adding any attributes found in them, but ignoring attributes already discovered in system/local. As a last resort, for any attributes not explicitly assigned at either the system or app level, it assigns default values from the file in the system/default directory.

When there's an app/user context, directory priority descends from user to app to system:

  • User directories -- highest priority.
  • App directories for currently running app (local, followed by default).
  • App directories for all other apps (local, followed by default) -- for exported settings only.
  • System directories (local, followed by default) -- lowest priority.

An attribute in savedsearches.conf, for example, might be set at all three levels: the user, the app, and the system. Splunk will always use the value of the user-level attribute, if any, in preference to a value for that same attribute set at the app or system level.

How app directory names affect precedence

Note: For most practical purposes, the information in this subsection probably won't matter, but it might prove useful if you need to force a certain order of evaluation or for troubleshooting.

To determine priority among the collection of apps directories, Splunk uses ASCII sort order. Files in an apps directory named "A" have a higher priority than files in an apps directory named "B", and so on. In addition, numbered directories have a higher priority than alphabetical directories and are evaluated in lexicographic, not numerical, order. For example, in descending order of precedence:

$SPLUNK_HOME/etc/apps/myapp1
$SPLUNK_HOME/etc/apps/myapp10
$SPLUNK_HOME/etc/apps/myapp2
$SPLUNK_HOME/etc/apps/myapp20
...
$SPLUNK_HOME/etc/apps/myappApple
$SPLUNK_HOME/etc/apps/myappBanana
$SPLUNK_HOME/etc/apps/myappZabaglione
...
$SPLUNK_HOME/etc/apps/myappapple
$SPLUNK_HOME/etc/apps/myappbanana
$SPLUNK_HOME/etc/apps/myappzabaglione
...

Note: When determining precedence in the app/user context, directories for the currently running app take priority over those for all other apps, independent of how they're named. Furthermore, other apps are only examined for exported settings.

Summary of directory precedence

Putting this all together, the order of directory priority, from highest to lowest, goes like this:

In the global context:

$SPLUNK_HOME/etc/system/local/*

$SPLUNK_HOME/etc/apps/A/local/* ... $SPLUNK_HOME/etc/apps/z/local/*

$SPLUNK_HOME/etc/apps/A/default/* ... $SPLUNK_HOME/etc/apps/z/default/*

$SPLUNK_HOME/etc/system/default/*

In the app/user context:

$SPLUNK_HOME/etc/users/*

$SPLUNK_HOME/etc/apps/Current_running_app/local/*

$SPLUNK_HOME/etc/apps/Current_running_app/default/*

$SPLUNK_HOME/etc/apps/A/local/*, $SPLUNK_HOME/etc/apps/A/default/*, ... $SPLUNK_HOME/etc/apps/z/local/*, $SPLUNK_HOME/etc/apps/z/default/* (but see note below)

$SPLUNK_HOME/etc/system/local/*

$SPLUNK_HOME/etc/system/default/*

Important: In the app/user context, all configuration files for the currently running app take priority over files from all other apps. This is true for the app's local and default directories. So, if the current context is app C, Splunk evaluates both $SPLUNK_HOME/etc/apps/C/local/* and $SPLUNK_HOME/etc/apps/C/default/* before evaluating the local or default directories for any other apps. Furthermore, Splunk only looks at configuration data for other apps if that data has been exported globally through the app's default.meta file, as described in this topic on setting app permissions.

Also, note that /etc/users/ is evaluated only when the particular user logs in or performs a search.

Example of how attribute precedence works

This example of attribute precedence uses props.conf. The props.conf file is unusual, because its context can be either global or app/user, depending on when Splunk is evaluating it. Splunk evaluates props.conf at both index time (global) and search time (apps/user).

Assume $SPLUNK_HOME/etc/system/local/props.conf contains this stanza:

[source::/opt/Locke/Logs/error*]
sourcetype = fatal-error

and $SPLUNK_HOME/etc/apps/t2rss/local/props.conf contains another version of the same stanza:

[source::/opt/Locke/Logs/error*]
sourcetype = t2rss-error
SHOULD_LINEMERGE = True
BREAK_ONLY_BEFORE_DATE = True

The line merging attribute assignments in t2rss always apply, as they only occur in that version of the file. However, there's a conflict with the sourcetype attribute. In the /system/local version, the sourcetype has a value of "fatal-error". In the /apps/t2rss/local version, it has a value of "t2rss-error".

Since this is a sourcetype assignment, which gets applied at index time, Splunk uses the global context for determining directory precedence. In the global context, Splunk gives highest priority to attribute assignments in system/local. Thus, the sourcetype attribute gets assigned a value of "fatal-error".

The final, internally merged version of the file looks like this:

[source::/opt/Locke/Logs/error*]
sourcetype = fatal-error
SHOULD_LINEMERGE = True
BREAK_ONLY_BEFORE_DATE = True

List of configuration files and their context

As mentioned, Splunk decides how to evaluate a configuration file based on the context that the file operates within, global or app/user. Generally speaking, files that affect data input, indexing, or deployment activities are global; files that affect search activities usually have a app/user context.

The props.conf and transforms.conf files can be evaluated in either a app/user or a global context, depending on whether Splunk is using them at index or search time.

Global configuration files

admon.conf
authentication.conf
authorize.conf
crawl.conf
deploymentclient.conf
distsearch.conf
indexes.conf
inputs.conf
outputs.conf
pdf_server.conf
procmonfilters.conf
props.conf -- global and app/user context
pubsub.conf
regmonfilters.conf
report_server.conf
restmap.conf
searchbnf.conf
segmenters.conf
server.conf
serverclass.conf
serverclass.seed.xml.conf
source-classifier.conf
sourcetypes.conf
sysmon.conf
tenants.conf
transforms.conf  -- global and app/user context
user_seed.conf -- special case: Must be located in /system/default
web.conf
wmi.conf

App/user configuration files

alert_actions.conf
app.conf
audit.conf
commands.conf
eventdiscoverer.conf
event_renderers.conf
eventtypes.conf
fields.conf
limits.conf
literals.conf
macros.conf
multikv.conf
props.conf -- global and app/user context
savedsearches.conf
tags.conf
times.conf
transactiontypes.conf
transforms.conf  -- global and app/user context
user-prefs.conf
workflow_actions.conf

Troubleshooting configuration precedence and other issues

Splunk's configuration file system supports many overlapping configuration files in many different locations. The price of this level of flexibility is that figuring out which value for which configuration option is being used in your Splunk installation can sometimes be quite complex. If you're looking for some tips on figuring out what configuration setting is being used in a given situation, read "Use btool to troubleshoot configurations" in the Troubleshooting Manual.

This documentation applies to the following versions of Splunk: 4.2.3 , 4.2.4 , 4.2.5 , 4.3 , 4.3.1 , 4.3.2 , 4.3.3 , 4.3.4 , 4.3.5 , 4.3.6 , 4.3.7 View the Article History for its revisions.


Comments

Those categories define the context in which the conf files operate. For authorize.conf, that context is global. It's true that you can use roles to restrict access to an app, but the role is still evaluated on a global basis.

Sgoodman, Splunker
October 11, 2012

I would say that the authorize.conf should be under the 'App/User Configuration Files' because you can create a role via an app - http://docs.splunk.com/Documentation/Splunk/latest/Developer/Step3AddConfigurations#Users_and_roles

Tyleraball
October 10, 2012

You must be logged into splunk.com in order to post comments. Log in now.

Was this documentation topic helpful?

If you'd like to hear back from us, please provide your email address:

We'd love to hear what you think about this topic or the documentation as a whole. Feedback you enter here will be delivered to the documentation team.

Feedback submitted, thanks!