Cascading knowledge bundle replication

In cascading bundle replication, the search head sends the knowledge bundle to a set of search peers, which then send the bundle to another set of search peers, and so on, until all peers receive the bundle. Compared to the classic method, the cascading method can achieve a massive increase in parallelization of the replication process, because it enlists search peers to help distribute the bundle.

Consider using cascading bundle replication when the number of search peers in the deployment exceeds 15 or 20.

How cascading bundle replication works

Terminology

A few concepts help to understand the process:

• Senders and receivers. Senders send the knowledge bundle; receivers receive the bundle. The search head that initiates the replication process serves only as a sender. Some search peers serve as both senders and receivers, in that they receive the bundle from the search head and also send the bundle onward to other search peers. Finally, the tertiary-level search peers function only as receivers.
• Cascading replication cycle. The replication cycle is the process that starts with the search head developing a replication plan and ends when all search peers have received and applied the bundle.
• Cascading replication plan. The search head creates a new execution plan for each bundle replication. The plan specifies the role of each search peer - whether it functions as a sender and, if so, which search peers it sends the bundle to. Each new replication plan can specify an entirely different set of sender search peers.
• Cascading replication payload. The replication payload is what the senders replicate to the receivers. It usually consists of a full or delta knowledge bundle.

The cascading replication cycle

A single cascading replication cycle proceeds like this:

1. The search head develops a replication plan and sends it directly to all the search peers. This plan identifies the search peers that will function as senders during the replication cycle. It also specifies the set of receivers for each sender search peer.

2. The search head sends the payload, consisting of the knowledge bundle, to its set of receivers.

3. Receivers that are also designated senders send the payload to their plan-specified set of receivers. Depending on the total number of search peers, there might be multiple intermediate levels of receiver/senders.

4. The process continues until all search peers receive the payload and report back to the search head.

Cascading replication is multisite indexer cluster aware. When replicating to a multisite indexer cluster, the search head picks one or more senders per site, and the senders then distribute the bundle to all search peers within their sites.

The following diagram shows a cascading knowledge bundle replication, with two layers of receiver/senders. By the end of the replication cycle, each search peer has a copy of the knowledge bundle.

The illustrated topology is specific to a single replication. The set of search peers that serve as senders changes with each new replication.

Configure cascading bundle replication

In addition to specifying the cascading policy, you must also specify a common security key for use by all search peers. There are several other settings available to fine tune the cascading process, both in distsearch.conf and server.conf, but use the defaults unless advised otherwise by Splunk Support.

Configure the cascading policy

To change the policy to cascading, edit the replicationPolicy setting in the [replicationSettings] stanza of distsearch.conf on the search head:

[replicationSettings]
 
replicationPolicy = cascading

You must restart the search head for the change to take effect.

Set the security key on all search peers

The search peers require a shared security key to communicate with each other. The key must be the same on each search peer.

This step is a requirement for cascading replication.

On each peer, edit the pass4SymmKey setting in the [cascading_replication] stanza of server.conf:

[cascading_replication]

pass4SymmKey = <password> 

You must restart each search peer for the key to take effect.

This pass4SymmKey is additional to any other pass4SymmKey also configured on the search peers for other purposes, such as indexer clustering communication or communication with a license manager.

Do not set this security key on the search head. The search head uses a different authorization method to communicate with its search peers.