Knowledge bundle replication overview

As part of the distributed search process, the search head periodically distributes its knowledge bundle to its search peers. This process is known as knowledge bundle replication.

What the knowledge bundle contains

The search peers use the search head's knowledge bundle to execute queries on its behalf. When executing a distributed search, the peers are ignorant of any local knowledge objects. They have access only to the objects in the search head's knowledge bundle.

Bundles typically contain a subset of files (configuration files and assets) from $SPLUNK_HOME/etc/system, $SPLUNK_HOME/etc/apps and $SPLUNK_HOME/etc/users on the search head.

The search head replicates either the full bundle or the delta since the last replication. The first time that the search head replicates a bundle to a new search peer, it replicates the full bundle. When the search head subsequently replicates a bundle to peers that have an earlier version of the bundle, it usually replicates only the bundle delta.

The process of knowledge bundle replication causes peers, by default, to receive nearly the entire contents of the search head's apps. If an app contains large binaries that do not need to be shared with the peers, you can eliminate them from the bundle and thus reduce the bundle size. See Modify the knowledge bundle.

Bundle size limit

The maxBundleSize setting under the [replicationSettings] stanza of distsearch.conf determines the maximum size for the bundle.

If a bundle exceeds the value of this setting, the search head will not push the bundle to the indexers. Searches on the indexers will instead use the previously pushed bundle, which will be within the size limit. This situation logs an error message and generates an admin-level message on the search head's Splunk Web.

To remediate, either increase the value of maxBundleSize or reduce the bundle size on the search head by eliminating unneeded files from the bundle. See Modify the knowledge bundle.

When you increase the maxBundleSize on the search head, you must also increase the max_content_length setting on the indexers by at least the same amount. Note that maxBundleSize is denoted in MB and max_content_length is denoted in bytes.

For example, if you increase maxBundleSize in distsearch.conf on the search head to 4GB :

[replicationSettings] 
maxBundleSize = 4096

you must also increase max_content_length in server.conf on the indexers (search peers) to at least 4GB:

[httpServer]
max_content_length =  4294967296

The knowledge bundle replication cycle

The replication cycle is the name given to the process of the search head distributing the latest knowledge bundle to its set of search peers. The details of the replication cycle vary depending on the particular replication policy. In particular, the cycle is more involved for the cascading policy, as described in The cascading replication cycle.

The search head replicates the knowledge bundle periodically in the background or when initiating a search.

Any additional time that search peers spend indexing lookup files, following bundle replication, occurs outside the replication cycle.

Only one replication cycle is active at a time. A new cycle does not start until an active one completes.

Location of the knowledge bundle

On the search head, the knowledge bundles resides under the $SPLUNK_HOME/var/run directory. The bundles have the extension .bundle for full bundles or .delta for delta bundles. They are tar files, so you can run tar tvf against them to see the contents.

During replication, the knowledge bundle gets distributed to the $SPLUNK_HOME/var/run/searchpeers directory on each search peer. Because the knowledge bundle reside in a different location on the search peers than on the search head, search scripts must not hardcode paths to resources.

Configure the knowledge bundle replication policy

There are several methods available for replicating knowledge bundles. These methods are embodied in replication policies:

• Classic. The search head replicates the bundle directly to all search peers. This is the default policy.
• Cascading. The search head replicates the bundle to several of its search peers. Those search peers then each replicate the bundle to a set of additional search peers, which might replicate the bundle to additional sets of peers, and so on, until all peers have received the bundle. By spreading the replication activity across multiple nodes, this policy improves performance for deployments with large numbers of search peers.
• Mounted. The search head places the bundle on shared storage, and the search peers each access the bundle from there. This policy is not recommended for general use.
• RFS. This method is not currently supported for Splunk Enterprise.

For detailed information on each available policy, see the topics that follow this one.

To select a policy, edit the replicationPolicy setting in the [replicationSettings] stanza of distsearch.conf on the search head:

[replicationSettings]
 
replicationPolicy = [classic | cascading | rfs | mounted]

For example:

[replicationSettings]
 
replicationPolicy = cascading

You must restart the search head for the change to take effect.

User authorization

All authorization for a distributed search originates from the search head. At the time it sends the search request to its search peers, the search head also distributes the authorization information. It tells the search peers the name of the user running the search, the user's role, and the location of the distributed authorize.conf file containing the authorization information.