Splunk® Enterprise

Distributed Search

Download manual as PDF

This documentation does not apply to the most recent version of Splunk. Click here for the latest version.
Download topic as PDF

Create a search head pool

To create a pool of search heads, follow these steps:

1. Set up a shared storage location accessible to each search head.

2. Configure each individual search head.

3. Stop the search heads.

4. Enable pooling on each search head.

5. Copy user and app directories to the shared storage location.

6. Restart the search heads.

The steps are described below in detail:

1. Set up a shared storage location accessible to each search head

So that each search head in a pool can share configurations and artifacts, they need to access a common set of files via shared storage:

  • On *nix platforms, set up an NFS mount.
  • On Windows, set up a CIFS (SMB) share.

Important: The Splunk user account needs read/write access to the shared storage location. When installing a search head on Windows, be sure to install it as a user with read/write access to shared storage. The Local System user does not have this access. For more information, see "Choose the user Splunk should run as" in the Installation manual.

2. Configure each search head

a. Set up each search head individually, specifying the search peers in the usual fashion. See "Add search peers to the search head".

b. Make sure that each search head has a unique serverName attribute, configured in server.conf. See "Manage distributed server names" for detailed information on this requirement. If the search head does not have a unique serverName, a warning will be generated at start-up. See "Warning about unique serverName attribute" for details.

c. Specify the necessary authentication. You have two choices:

  • Specify user authentication on each search head separately. A valid user on one search head is not automatically a user on another search head in the pool. You can use LDAP to centrally manage user authentication, as described in "Set up user authentication with LDAP".
  • Place a common authentication configuration on shared storage, to be used by all pool members. You must restart the pool members after any change to the authentication.

Note: Any authentication change made on an individual pool member (for example, via Splunk Web) overrides for that pool member only any configuration on shared storage. You should, therefore, generally avoid making authentication changes through Splunk Web if a common configuration already exists on shared storage.

3. Stop the search heads

Before enabling pooling, you must stop splunkd. Do this for each search head in the pool.

4. Enable pooling on each search head

Use the CLI command splunk pooling enable to enable pooling on a search head. The command sets certain values in server.conf. It also creates subdirectories within the shared storage location and validates that Splunk Enterprise can create and move files within them.

Here's the command syntax:

  splunk pooling enable <path_to_shared_storage> [--debug]

Note:

  • On NFS, <path_to_shared_storage> should be the NFS's share mountpoint.
  • On Windows, <path_to_shared_storage> should be the UNC path of the CIFS/SMB share.
  • The --debug parameter causes the command to log additional information to btool.log.

Execute this command on each search head in the pool.

The command sets values in the [pooling] stanza of the server.conf file in $SPLUNK_HOME/etc/system/local.

You can also directly edit the [pooling] stanza of server.conf. For detailed information on server.conf, look here.

Important: The [pooling] stanza must be placed in the server.conf file directly under $SPLUNK_HOME/etc/system/local/. This means that you cannot deploy the [pooling] stanza via an app, either on local disk or on shared storage. For details see the server.conf spec file.

5. Copy user and app directories to the shared storage location

Copy the contents of the $SPLUNK_HOME/etc/apps and $SPLUNK_HOME/etc/users directories on an existing search head into the empty /etc/apps and /etc/users directories in the shared storage location. Those directories were created in step 4 and reside under the <path_to_shared_storage> that you specified at that time.

For example, if your NFS mount is at /tmp/nfs, copy the apps subdirectories that match this pattern:

$SPLUNK_HOME/etc/apps/*

into

/tmp/nfs/etc/apps

This results in a set of subdirectories like:

/tmp/nfs/etc/apps/search
/tmp/nfs/etc/apps/launcher
/tmp/nfs/etc/apps/unix
[...]

Similarly, copy the user subdirectories:

$SPLUNK_HOME/etc/users/*

into

/tmp/nfs/etc/users

Important: You can choose to copy over just a subset of apps and user subdirectories; however, be sure to move them to the precise locations described above.

6. Restart the search heads

After running the splunk pooling enable command, restart splunkd. Do this for each search head in the pool.

PREVIOUS
Overview of search head pooling
  NEXT
Use a load balancer with the search head pool

This documentation applies to the following versions of Splunk® Enterprise: 6.0, 6.0.1, 6.0.2, 6.0.3, 6.0.4, 6.0.5, 6.0.6, 6.0.7, 6.0.8, 6.0.9, 6.0.10, 6.0.11, 6.0.12, 6.0.13, 6.0.14, 6.0.15, 6.1, 6.1.1, 6.1.2, 6.1.3, 6.1.4, 6.1.5, 6.1.6, 6.1.7, 6.1.8, 6.1.9, 6.1.10, 6.1.11, 6.1.12, 6.1.13, 6.1.14, 6.2.0, 6.2.1, 6.2.2, 6.2.3, 6.2.4, 6.2.5, 6.2.6, 6.2.7, 6.2.8, 6.2.9, 6.2.10, 6.2.11, 6.2.12, 6.2.13, 6.2.14, 6.2.15, 6.3.0, 6.3.1, 6.3.2, 6.3.3, 6.3.4, 6.3.5, 6.3.6, 6.3.7, 6.3.8, 6.3.9, 6.3.10, 6.3.11, 6.3.12, 6.3.13, 6.4.0, 6.4.1, 6.4.2, 6.4.3, 6.4.4, 6.4.5, 6.4.6, 6.4.7, 6.4.8, 6.4.9, 6.4.10, 6.4.11, 6.5.0, 6.5.1, 6.5.1612 (Splunk Cloud only), 6.5.2, 6.5.3, 6.5.4, 6.5.5, 6.5.6, 6.5.7, 6.5.8, 6.5.9, 6.5.10, 6.6.0, 6.6.1, 6.6.2, 6.6.3, 6.6.4, 6.6.5, 6.6.6, 6.6.7, 6.6.8, 6.6.9, 6.6.10, 6.6.11, 6.6.12, 7.0.0, 7.0.1, 7.0.2, 7.0.3, 7.0.4, 7.0.5, 7.0.6, 7.0.7, 7.0.8, 7.0.9, 7.0.10, 7.0.11, 7.1.0, 7.1.1, 7.1.2, 7.1.3, 7.1.4, 7.1.5, 7.1.6, 7.1.7, 7.1.8, 7.1.9, 7.2.0, 7.2.1, 7.2.2, 7.2.3, 7.2.4, 7.2.5, 7.2.6, 7.2.7, 7.2.8, 7.3.0, 7.3.1, 7.3.2


Comments

Tphi - You can theoretically remove the apps directory on the individual search heads and searching will continue to function normally. However, if at some later time you need to break the pool (for example, for maintenance or troubleshooting), the search head will be in a strange state, missing some essential files such as the search app and some configuration files. You could then work around that problem by copying over the app directory from some other location such as a backup or the NFS mount (assuming the shared storage is still available and wasn't the reason you initiated troubleshooting), but is the savings in disk space really worth the potential disruption and additional effort?

Sgoodman
January 30, 2014

Since the content of $SPLUNK_HOME/etc/apps/ (and users) is copied to SH_POOL shared nfs, do we need to keep the $SPLUNK_HOME/etc/apps for any reason?

Tphi
January 28, 2014

Was this documentation topic helpful?

Enter your email address, and someone from the documentation team will respond to you:

Please provide your comments here. Ask a question or make a suggestion.

You must be logged into splunk.com in order to post comments. Log in now.

Please try to keep this discussion focused on the content covered in this documentation topic. If you have a more general question about Splunk functionality or are experiencing a difficulty with Splunk, consider posting a question to Splunkbase Answers.

0 out of 1000 Characters