Mounted knowledge bundle replication

The mounted policy is not recommended for general use. Instead, use the classic or the cascading policy.

As a result of changes to bundle replication made in the 5.0 timeframe, such as the introduction of delta-based replication and improvements in streaming, the practical use case for mounted bundles is now extremely limited. In most cases, mounted bundles make little difference in the amount of network traffic or the speed at which bundle changes get distributed to the search peers. At the same time, they add significant management complexity, particularly when combined with shared storage. Because of delta-based replication, even if your configurations contain large files, normal bundle replication entails little ongoing replication cost, as long as those files rarely change.

By default, the search head replicates and distributes the knowledge bundle to each search peer. You can instead tell the search peers to mount the knowledge bundle's directory location, eliminating the need for bundle replication. When you mount a knowledge bundle on shared storage, it's referred to as a mounted bundle.

Most shared storage solutions don't work well across a WAN. Since mounted bundles require shared storage, you generally should not implement them across a WAN.

Mounted bundle architectures

Depending on your search head configuration, there are a number of ways to set up mounted bundles. These are some of the typical ones:

• For a single search head. Mount the knowledge bundle on shared storage. All the search peers then access the bundle to process search requests. This diagram illustrates a single search head with a mounted bundle on shared storage:

• For multiple non-clustered search heads. Maintain the knowledge bundle(s) on each search head's local storage. In this diagram, each search head maintains its own bundle, which each search peer mounts and accesses individually:

In each case, the search peers need access to each search head's $SPLUNK_HOME/etc/{apps,users,system} subdirectories.

The search peers use the mounted directories only when fulfilling the search head's search requests. For indexing and other purposes not directly related to distributed search, the search peers will use their own, local apps, users, and system directories, the same as any other indexer.

Configure mounted bundles

To set up mounted bundles, you need to configure both the search head and its search peers. The procedures described here assume the bundles are on shared storage, but they do not need to be. They just need to be in some location that both the search head and its search peers can access.

Configure the search head

Here are the steps you take on the search head:

1. Mount the bundle subdirectories ($SPLUNK_HOME/etc/{apps,users,system}) on shared storage. The simplest way to do this is to mount the search head's entire $SPLUNK_HOME/etc directory:

• On *nix platforms, set up an NFS mount.

• On Windows, set up a CIFS (SMB) share.

The search head's Splunk user account needs read/write access to the shared storage location. The search peers must have only read access to the bundle subdirectories, to avoid file-lock issues. Search peers do not need to update any files in the shared storage location.

2. To change the policy to mounted, edit the replicationPolicy setting in distsearch.conf on the search head:

[replicationSettings]
 
replicationPolicy = mounted

3. Restart the search head.

Configure the search peers

For each search peer, follow these steps to access the mounted bundle:

1. Mount the bundle directory on the search peer.

2. Create a distsearch.conf file in $SPLUNK_HOME/etc/system/local/ on the search peer. For each search head that the peer is connected to, create a [searchhead:<searchhead-splunk-server-name>] stanza, with these attributes:

[searchhead:<searchhead-splunk-server-name>]
mounted_bundles=true
bundles_location=<path_to_bundles>

Note the following:

• The search peer's configuration file must contain only the [searchhead:<searchhead-splunk-server-name>] stanza(s). The other stanzas in distsearch.conf are for search heads only.

• To identify the <searchhead-splunk-server-name>, run this command on the search head:

   splunk show servername 

• If the search peer is running against a search head cluster, the [searchhead:] stanza on the peer must specify the cluster's GUID, not the server name of any cluster members. For example:

   [searchhead:C7729EE6-D260-4268-A699-C1F95AAD07D5]

To identify the GUID, run this command on a cluster member:

   splunk show shcluster-status 

The cluster GUID is the value of the id field, located in the captain section of the results.

• The <path_to_bundles> needs to specify the mountpoint on the search peer, not on the search head. For example, say $SPLUNK_HOME on your search head is /opt/splunk, and you export /opt/splunk/etc via NFS. Then, on the search peer, you mount that NFS share at /mnt/splunk-head. The value of <path_to_bundles> should be /mnt/splunk-head, not /opt/splunk.

• If multiple non-clustered search heads will be distributing searches to this search peer, you must create a separate stanza on the search peer for each of them.

3. Restart the search peer.

You can optionally set up symbolic links to the bundle subdirectories (apps,users,system) to ensure that the search peer has access only to the necessary subdirectories in the search head's /etc directory. See the following example for details on how to do this.

Example configuration

Here's an example of how to set up mounted bundles on shared storage:

Search head:

On a search head whose Splunk Enterprise server name is "searcher01":

1. Mount the search head's $SPLUNK_HOME/etc directory to shared storage with read/write access.

2. In the distsearch.conf file on the search head, change the replication policy to mounted:

[replicationSettings]
 
replicationPolicy = mounted

3. Restart the search head.

Search peers:

For each search peer:

1. Mount the search head's $SPLUNK_HOME/etc directory on the search peer to:

    /mnt/searcher01

2. (Optional.) Create a directory that consists of symbolic links to the bundle subdirectories:

   /opt/shared_bundles/searcher01
   /opt/shared_bundles/searcher01/system -> /mnt/searcher01/system
   /opt/shared_bundles/searcher01/users -> /mnt/searcher01/users
   /opt/shared_bundles/searcher01/apps -> /mnt/searcher01/apps

Note: This optional step is useful for ensuring that the peer has access only to the necessary subdirectories.

3. Create a distsearch.conf file in $SPLUNK_HOME/etc/system/local/ on the search peer, with this stanza:

[searchhead:searcher01]
mounted_bundles = true
bundles_location = /opt/shared_bundles/searcher01

4. Restart the search peer.

5. Repeat the process for each search peer.