How authorization works in distributed searches

The authorization settings that a search peer uses when processing distributed searches are different from those that it uses for its local activities, such as administration and local search requests:

• When processing a distributed search, the search peer uses the settings contained in the knowledge bundle that the search head distributes to all the search peers when it sends them a search request. These settings are created and managed on the search head.
• When performing local activities, the search peer uses the authorization settings created and stored locally on the search peer itself.

When managing distributed searches, it is therefore important that you distinguish between these two types of authorization. You need to be particularly aware of how authorization settings get distributed through the knowledge bundle when you're managing a system with search head pooling or mounted bundles.

For background information, read about these key concepts:

• Splunk Enterprise authorization: The topic "About role-based user access" in the Securing Splunk Enterprise manual
• Mounted bundles: The chapter "Mount the knowledge bundle" in this manual
• Search head pooling: The chapter "Search head pooling" in this manual

Manage authorization for distributed searches

All authorization settings are stored in one or more authorize.conf files. This includes settings configured through Splunk Web or the CLI. It is these authorize.conf files that get distributed from the search head to the search peers. On the knowledge bundle, the files are usually located in either /etc/system/{local,default} and/or /etc/apps/<app-name>/{local,default}.

Since search peers automatically use the settings in the knowledge bundle, things normally work fine. You configure roles for your users on the search head, and the search head automatically distributes those configurations to the search peers when it distributes the search itself.

With search head pooling, however, you must take care to ensure that the search heads and the search peers all use the same set of authorize.conf file(s). For this to happen, you must make sure:

• All search heads in the pool use the same set of authorize.conf files

• The set of authorize.conf files that the search heads use goes into the knowledge bundle so that they get distributed to the search peers.

This topic describes the four main scenarios, based on whether or not you're using search head pooling or mounted bundles. It describes the scenarios in order from simple to complex.

Four scenarios

What you need to do with the distributed search authorize.conf files depends on whether your deployment implements search head pooling or mounted bundles. The four scenarios are:

• No search head pooling, no mounted bundles
• No search head pooling, mounted bundles
• Search head pooling, no mounted bundles
• Search head pooling, mounted bundles

The first two scenarios "just work" but the last two scenarios require careful planning. For the sake of completeness, this section describes all four scenarios.

Note: These scenarios address authorization settings for distributed search only. Local authorization settings function the same independent of your distributed search deployment.

No search head pooling, no mounted bundles

Whatever authorization settings you have on the search head get automatically distributed to its search peers as part of the replicated knowledge bundle that they receive with distributed search requests.

No search head pooling, mounted bundles

Whatever authorization settings you have on the search head get automatically placed in the mounted bundle and used by the search peers during distributed search processing.

Search head pooling, no mounted bundles

The search heads in the pool share their /apps and /users directories but not their /etc/system/local directories. Any authorize.conf file in an /apps subdirectory will be automatically shared by all search heads and included in the knowledge bundle when any of the search heads distributes a search request to the search peers.

The problem arises because authorization changes can also get saved to an authorize.conf file in a search head's /etc/system/local directory (for example, if you update the search head's authorization settings via Splunk Web). This directory does not get shared among the search heads in the pool, but it still gets distributed to the search peers as part of the knowledge bundle. Because of how the configuration system works, any copy of authorize.conf file in /etc/system/local will have precedence over a copy in an /apps subdirectory. (See "Configuration file precedence" in the Admin manual for details.)

Therefore, a copy of authorize.conf that gets distributed to the search peers from a single search head's /etc/system/local directory has precedence over any copies distributed from the search head pool's shared directory. Unless you account for this situation, the search peers can end up using different authorization settings for different searches, depending on which search head distributed the search to them. For most situations, this is not what you want to occur.

To avoid this problem, you need to make sure that any changes made to a search head's /etc/system/local/authorize.conf file get propagated to all search heads in the pool. One way to handle this is to move any changed /etc/system/local/authorize.conf file into an app subdirectory, since all search heads in the pool share the /apps directory.

Search head pooling, mounted bundles

This is similar to the previous scenario. The search heads in the pool share their /apps and /users directories but not their /etc/system/local directories. Any authorize.conf file in an /apps subdirectory will be automatically shared by all search heads. It will also be included in the mounted bundle that the search peers use when processing a search request from any of the search heads.

However, authorization changes can also wind up in an authorize.conf file in a search head's /etc/system/local directory (for example, if you update the search head's authorization settings via Splunk Web). This directory does not get automatically shared among the search heads in the pool. It also does not get automatically distributed to the mounted bundle that the search peers use. Therefore, you must provide some mechanism that ensures that all the search heads and all the search peers have access to that version of authorize.conf.

The simplest way to handle this is to move any changed /etc/system/local/authorize.conf file into an app subdirectory, since both the pooled search heads and all the search peers share the /apps directory.