Splunk® Enterprise

Distributed Deployment Manual

Download manual as PDF

Splunk version 4.x reached its End of Life on October 1, 2013. Please see the migration information.
This documentation does not apply to the most recent version of Splunk. Click here for the latest version.
Download topic as PDF

Mount the knowledge bundle

The set of data that a search head distributes to its search peers is called the knowledge bundle. For information on the contents and purpose of this bundle, see "What search heads send to search peers".

The bundle contents reside in the search head's $SPLUNK_HOME/etc/{apps,users,system} subdirectories.

By default, the search head replicates and distributes the knowledge bundle to each search peer. For greater efficiency, you can instead tell the search peers to mount the knowledge bundle's directory location, eliminating the need for bundle replication. When you mount a knowledge bundle, it's referred to as a mounted bundle.

Why use mounted bundles

Mounted bundles are useful if you have large amounts of search-time data, which could otherwise slow the replication process. One common cause of slow bundle replication is large lookup tables.

Splunk writes a warning in splunkd.log if bundle replication is taking a long time. For example:

DistributedBundleReplicationManager - bundle replication to 2 peer(s) took 
too long (12943ms), bundle file size=296150KB ...

Mounted bundle architectures

Depending on your search head configuration, there are a number of ways to set up mounted bundles. These are some of the typical ones:

  • For a single search head. Mount the knowledge bundle on shared storage. All the search peers then access the bundle to process search requests. This diagram illustrates a single search head with a mounted bundle on shared storage:
       Mounted bundles 3.jpg

  • For multiple pooled search heads. For multiple search heads, you can combine mounted bundles with search head pooling. The pooled search heads maintain one bundle on the shared storage, and all search peers access that bundle. This diagram shows search head pooling with a mounted bundle:
       Mounted bundles 1.jpg

  • For multiple non-pooled search heads. Maintain the knowledge bundle(s) on each search head's local storage (no search head pooling). In this diagram, each search head maintains its own bundle, which each search peer mounts and accesses individually:
       Mounted bundles 2.jpg

There are numerous other architectures you can design with mounted bundles. You could, for example, use shared storage for multiple search heads, but without search head pooling. On the shared storage, you would maintain separate bundles for each search head. The search peers would need to access each bundle individually.

In each case, the search peers need access to each search head's $SPLUNK_HOME/etc/{apps,users,system} subdirectories. In the case of search head pooling, the search peers need access to the pool's shared set of subdirectories.

Important: The search peers use the mounted directories only when fulfilling the search head's search requests. For indexing and other purposes not directly related to distributed search, the search peers will use their own, local apps, users, and system directories, the same as any other indexer.

Configure mounted bundles

To set up mounted bundles, you need to configure both the search head and its search peers. The procedures described here assume the bundles are on shared storage, but they do not need to be. They just need to be in some location that both the search head and its search peers can access.

Note: It's best not to locate mounted bundles in the search head's local $SPLUNK_HOME path.

These procedures also assume a single search head (no search head pooling). For details on how to configure mounted bundles with search head pooling, see "Use mounted bundles with search head pooling" below.

Configure the search head

Here are the steps you take on the search head:

1. Mount the bundle subdirectories ($SPLUNK_HOME/etc/{apps,users,system}) on shared storage. The simplest way to do this is to mount the search head's entire $SPLUNK_HOME/etc directory:

  • On *nix platforms, set up an NFS mount.
  • On Windows, set up a CIFS (SMB) share.

Important: The search head's Splunk user account needs read/write access to the shared storage location. The search peers need read access to the bundle subdirectories.

2. In the distsearch.conf file on the search head, set:


This stops the search head from replicating bundles to the search peers.

3. Restart the search head.

Configure the search peers

For each search peer, follow these steps to access the mounted bundle:

1. Mount the bundle directory on the search peer.

2. Create a distsearch.conf file in $SPLUNK_HOME/etc/system/local/ on the search peer. For each search head that the peer is connected to, create a [searchhead:<searchhead-splunk-server-name>] stanza, with these attributes:


Note the following:

  • The search peer's configuration file must contain only the [searchhead:<searchhead-splunk-server-name>] stanza(s). The other stanzas in distsearch.conf are for search heads only.
  • To identify the <searchhead-splunk-server-name>, run this command on the search head:
   splunk show servername 
  • The <path_to_bundles> needs to specify the mountpoint on the search peer, not on the search head. For example, say $SPLUNK_HOME on your search head is /opt/splunk, and you export /opt/splunk/etc via NFS. Then, on the search peer, you mount that NFS share at /mnt/splunk-head. The value of <path_to_bundles> should be /mnt/splunk-head, not /opt/splunk.

Important: If multiple search heads will be distributing searches to this search peer, you must create a separate stanza on the search peer for each of them. This is necessary even if you're using search head pooling.

3. Restart the search peer.

Note: You can optionally set up symbolic links to the bundle subdirectories (apps,users,system) to ensure that the search peer has access only to the necessary subdirectories in the search head's /etc directory. See the following example for details on how to do this.

Example configuration

Here's an example of how to set up mounted bundles on shared storage:

Search head

On a search head whose Splunk server name is "searcher01":

1. Mount the search head's $SPLUNK_HOME/etc directory to shared storage with read/write access.

2. In the distsearch.conf file on the search head, set:

shareBundles = false

3. Restart the search head.

Search peers

For each search peer:

1. Mount the search head's $SPLUNK_HOME/etc directory on the search peer to:


2. (Optional.) Create a directory that consists of symbolic links to the bundle subdirectories:

   /opt/shared_bundles/searcher01/system -> /mnt/searcher01/system
   /opt/shared_bundles/searcher01/users -> /mnt/searcher01/users
   /opt/shared_bundles/searcher01/apps -> /mnt/searcher01/apps

Note: This optional step is useful for ensuring that the peer has access only to the necessary subdirectories.

3. Create a distsearch.conf file in $SPLUNK_HOME/etc/system/local/ on the search peer, with this stanza:

mounted_bundles = true
bundles_location = /opt/shared_bundles/searcher01

4. Restart the search peer.

5. Repeat the process for each search peer.

Use mounted bundles with search head pooling

The process for configuring mounted bundles is basically no different if you're using search head pooling to manage multiple search heads. A few things to keep in mind:

  • Use the same shared storage location for both the search head pool and the mounted bundles. Search head pooling uses a subset of the directories required for mounted bundles.
  • Search head pooling itself only requires that you mount the $SPLUNK_HOME/etc/{apps,users} directories. However, when using mounted bundles, you must also provide a mounted $SPLUNK_HOME/etc/system directory. This doesn't create any conflict among the search heads, as they will always use their own versions of the system directory and ignore the mounted version.
  • The search peers must create separate stanzas in distsearch.conf for each search head in the pool. The bundles_location in each of those stanzas must be identical.

See "Configure search head pooling" for information on setting up a search head pool.

Example configuration: Search head pooling with mounted bundles

This example shows how to combine search head pooling and mounted bundles in one system. There are two main sections to the example:

1. Set up a search head pool consisting of two search heads. In this part, you also mount the bundles.

2. Set up the search peers so that they can access bundles from the search head pool.

The example assumes you're using an NFS mount for the shared storage location.

Part 1: Set up the search head pool

Before configuring the pool, perform these preliminary steps:

1. Enable two Splunk instances as search heads. This example assumes that the instances are named "searcher01" and "searcher02".

2. Set up a shared storage location accessible to each search head. This example assumes that you set up an NFS mountpoint, specified on the search heads as /mnt/search-head-pooling.

For detailed information on these steps, see "Create a pool of search heads".

Now, configure the search head pool:

1. On each search head, stop splunkd:

splunk stop splunkd

2. On each search head, enable search head pooling. In this example, you're using an NFS mount of /mnt/search-head-pooling as your shared storage location:

splunk pooling enable /mnt/search-head-pooling [--debug]

Among other things, this step creates empty /etc/apps and /etc/users directories under /mnt/search-head-pooling. Step 3 uses those directories.

3. Copy the contents of the $SPLUNK_HOME/etc/apps and $SPLUNK_HOME/etc/users directories on one of the search heads into the /etc/apps and /etc/users subdirectories under /mnt/search-head-pooling:

cp -r $SPLUNK_HOME/etc/apps/* /mnt/search-head-pooling/etc/apps

cp -r $SPLUNK_HOME/etc/users/* /mnt/search-head-pooling/etc/users

4. Copy one search head's $SPLUNK_HOME/etc/system directory to /mnt/search-head-pooling/etc/system.

cp -r $SPLUNK_HOME/etc/system /mnt/search-head-pooling/etc/

5. On each search head, edit the distsearch.conf file to set shareBundles = false:

shareBundles = false

6. On each search head, start splunkd:

splunk start splunkd

Your search head pool should now be up and running.

Part 2: Mount bundles on the search peers

Now, mount the bundles on the search peers.

On each search peer, perform these steps:

1. Mount the shared storage location (the same location that was earlier set to /mnt/search-head-pooling on the search heads) so that it appears as /mnt/bundles on the peer.

2. Create a directory that consists of symbolic links to the bundle subdirectories:

   /opt/shared_bundles/bundles/system -> /mnt/bundles/etc/system
   /opt/shared_bundles/bundles/users -> /mnt/bundles/etc/users
   /opt/shared_bundles/bundles/apps -> /mnt/bundles/etc/apps

3. Create a distsearch.conf file in $SPLUNK_HOME/etc/system/local/ on the search peer, with stanzas for each of the two search heads:

mounted_bundles = true
bundles_location = /opt/shared_bundles/bundles

mounted_bundles = true
bundles_location = /opt/shared_bundles/bundles

4. Restart the search peer:

splunk restart splunkd

Repeat the process for each search peer.

Configure distributed search
Configure search head pooling

This documentation applies to the following versions of Splunk® Enterprise: 4.3, 4.3.1, 4.3.2, 4.3.3, 4.3.4, 4.3.5, 4.3.6, 4.3.7


There's a section in this topic that talks about how to configure mounted bundles when also employing search head pooling. Basically, with a few caveats (as described there), you configure mounted bundles the same way, whether or not you've got search head pooling.

October 5, 2011

for mounted bundles + search head pooling, you don't have to tell the searchheads to stop sending out the knowledge bundles in some way? or are you supposed to follow the directions for mounted bundles and then also apply the directions for mounted bundles + search head pooling?

September 26, 2011

Was this documentation topic helpful?

Enter your email address, and someone from the documentation team will respond to you:

Please provide your comments here. Ask a question or make a suggestion.

You must be logged into splunk.com in order to post comments. Log in now.

Please try to keep this discussion focused on the content covered in this documentation topic. If you have a more general question about Splunk functionality or are experiencing a difficulty with Splunk, consider posting a question to Splunkbase Answers.

0 out of 1000 Characters