Knowledge bundle replication overview
As part of the distributed search process, the search head periodically distributes its knowledge bundle to its search peers. This process is known as knowledge bundle replication.
What the knowledge bundle contains
The search peers use the search head's knowledge bundle to execute queries on its behalf. When executing a distributed search, the peers are ignorant of any local knowledge objects. They have access only to the objects in the search head's knowledge bundle.
Bundles typically contain a subset of files (configuration files and assets) from $SPLUNK_HOME/etc/system
, $SPLUNK_HOME/etc/apps
and $SPLUNK_HOME/etc/users
on the search head.
The search head replicates either the full bundle or the delta since the last replication. The first time that the search head replicates a bundle to a new search peer, it replicates the full bundle. When the search head subsequently replicates a bundle to peers that have an earlier version of the bundle, it usually replicates only the bundle delta.
The process of knowledge bundle replication causes peers, by default, to receive nearly the entire contents of the search head's apps. If an app contains large binaries that do not need to be shared with the peers, you can eliminate them from the bundle and thus reduce the bundle size. See Modify the knowledge bundle.
Bundle size limit
The maxBundleSize
setting under the [replicationSettings]
stanza of distsearch.conf
determines the maximum size for the bundle.
If a bundle exceeds the value of this setting, the search head will not push the bundle to the indexers. Searches on the indexers will instead use the previously pushed bundle, which will be within the size limit. This situation logs an error message and generates an admin-level message on the search head's Splunk Web.
To remediate, either increase the value of maxBundleSize
or reduce the bundle size on the search head by eliminating unneeded files from the bundle. See Modify the knowledge bundle.
When you increase the maxBundleSize
on the search head, you must also increase the max_content_length
setting on the indexers by at least the same amount. Note that maxBundleSize
is denoted in MB and max_content_length
is denoted in bytes.
For example, if you increase maxBundleSize
in distsearch.conf
on the search head to 4GB :
[replicationSettings] maxBundleSize = 4096
you must also increase max_content_length
in server.conf
on the indexers (search peers) to at least 4GB:
[httpServer] max_content_length = 4294967296
The knowledge bundle replication cycle
The replication cycle is the name given to the process of the search head distributing the latest knowledge bundle to its set of search peers. The details of the replication cycle vary depending on the particular replication policy. In particular, the cycle is more involved for the cascading policy, as described in The cascading replication cycle.
The search head replicates the knowledge bundle periodically in the background or when initiating a search.
Any additional time that search peers spend indexing lookup files, following bundle replication, occurs outside the replication cycle.
Only one replication cycle is active at a time. A new cycle does not start until an active one completes.
Location of the knowledge bundle
On the search head, the knowledge bundles resides under the $SPLUNK_HOME/var/run
directory. The bundles have the extension .bundle
for full bundles or .delta
for delta bundles. They are tar files, so you can run tar tvf
against them to see the contents.
During replication, the knowledge bundle gets distributed to the $SPLUNK_HOME/var/run/searchpeers
directory on each search peer. Because the knowledge bundle reside in a different location on the search peers than on the search head, search scripts must not hardcode paths to resources.
Configure the knowledge bundle replication policy
There are several methods available for replicating knowledge bundles. These methods are embodied in replication policies:
- Classic. The search head replicates the bundle directly to all search peers. This is the default policy.
- Cascading. The search head replicates the bundle to several of its search peers. Those search peers then each replicate the bundle to a set of additional search peers, which might replicate the bundle to additional sets of peers, and so on, until all peers have received the bundle. By spreading the replication activity across multiple nodes, this policy improves performance for deployments with large numbers of search peers.
- Mounted. The search head places the bundle on shared storage, and the search peers each access the bundle from there. This policy is not recommended for general use.
- RFS. This method is not currently supported for Splunk Enterprise.
For detailed information on each available policy, see the topics that follow this one.
To select a policy, edit the replicationPolicy
setting in the [replicationSettings]
stanza of distsearch.conf
on the search head:
[replicationSettings] replicationPolicy = [classic | cascading | rfs | mounted]
For example:
[replicationSettings] replicationPolicy = cascading
You must restart the search head for the change to take effect.
User authorization
All authorization for a distributed search originates from the search head. At the time it sends the search request to its search peers, the search head also distributes the authorization information. It tells the search peers the name of the user running the search, the user's role, and the location of the distributed authorize.conf
file containing the authorization information.
What search heads send to search peers | Modify the knowledge bundle |
This documentation applies to the following versions of Splunk® Enterprise: 8.1.0, 8.1.1, 8.1.2, 8.1.3, 8.1.4, 8.1.5, 8.1.6, 8.1.7, 8.1.8, 8.1.9, 8.1.10, 8.1.11, 8.1.12, 8.1.13, 8.1.14, 8.2.0, 8.2.1, 8.2.2, 8.2.3, 8.2.4, 8.2.5, 8.2.6, 8.2.7, 8.2.8, 8.2.9, 8.2.10, 8.2.11, 8.2.12, 9.0.0, 9.0.1, 9.0.2, 9.0.3, 9.0.4, 9.0.5, 9.0.6, 9.0.7, 9.0.8, 9.0.9, 9.0.10, 9.1.0, 9.1.1, 9.1.2, 9.1.3, 9.1.4, 9.1.5, 9.1.6, 9.2.0, 9.2.1, 9.2.2, 9.2.3, 9.3.0, 9.3.1
Feedback submitted, thanks!