Deploy a search head cluster

This topic covers the key steps needed to configure and start a search head cluster.

Parts of a search head cluster

A search head cluster consists of a group of search heads that share configurations, job scheduling, and search artifacts. The search heads are known as the cluster members.

One cluster member has the role of captain, which means that it coordinates job and replication activities among all the members. It also serves as a search head like any other member, running search jobs, serving results, and so on. Over time, the role of captain can shift among the cluster members.

In addition to the set of search head members that constitute the actual cluster, a functioning cluster requires several other components:

• The deployer. This is a Splunk Enterprise instance that distributes apps and other configurations to the cluster members. It stands outside the cluster and cannot run on the same instance as a cluster member. It can, however, under some circumstances, reside on the same instance as other Splunk Enterprise components, such as a deployment server or an indexer cluster manager node.
• Search peers. These are the indexers that cluster members run their searches across. The search peers can be either independent indexers or nodes in an indexer cluster.
• Load balancer. This is third-party software or hardware optionally residing between the users and the cluster members. With a load balancer in place, users can access the set of search heads through a single interface, without needing to specify a particular one.

This diagram of a small search head cluster, consisting of three members, illustrates the various components and their relationships:

This topic focuses on setting up the cluster members and the deployer. Other topics in this chapter describe how to configure search peers, connect with an indexer cluster, and add a load balancer.

Deploy the cluster

These are the key steps in deploying clusters:

1. Identify your requirements.

2. Set up the deployer.

3. Install the Splunk Enterprise instances.

4. Initialize cluster members.

5. Bring up the cluster captain.

6. Perform post-deployment set-up.

1. Identify your requirements

a. Determine the cluster size, that is, the number of search heads that you want to include in it. It usually makes sense to put all your search heads in a single cluster. Factors that influence cluster size include the anticipated search load and number of concurrent users, and your availability and failover needs. See "About search head clustering".

b. Decide what replication factor you want to implement. The replication factor is the number of copies of search artifacts that the cluster maintains. Your optimal replication factor depends on factors specific to your environment, but essentially involves a trade-off between failure tolerance and storage capacity. A higher replication factor means that more copies of the search artifacts will reside on more cluster members, so your cluster can tolerate more member failures without needing to use a proxy to access the artifacts. But it also means that you will need more storage to handle the additional copies. See "Choose the replication factor for the search head cluster."

c. Determine whether the search head cluster will be running against a group of standalone indexers or an indexer cluster. For information on indexer clusters, see "About indexer clusters and index replication" in the Managing Indexers and Clusters of Indexers manual.

d. Study the topic "System requirements and other deployment considerations for search head clusters" for information on other key issues.

2. Set up the deployer

It is recommended that you select the deployer now, as part of cluster set-up, because you need a deployer in place before you can distribute apps and updated configurations to the cluster members.

a. Choose a Splunk Enterprise instance for the deployer functionality.

This instance cannot be a member of the search head cluster, but, under some circumstances, it can be a Splunk Enterprise instance in use for other purposes. If necessary, install a new Splunk Enterprise instance to serve as the deployer. See "Deployer requirements".

If you have multiple clusters, you must use a separate deployer for each cluster, unless you are deploying identical configurations across all the clusters. See "Deploy to multiple clusters."

Deployer functionality is automatically enabled on all Splunk Enterprise instances. The main configuration step is to specify the deployer's security key, as described in the next step. Later in the deployment process, you point the cluster members at this deployer instance, so that they have access to it.

For information on how to use the deployer to distribute apps to cluster members, see "Use the deployer to distribute apps and configuration updates."

b. Configure the deployer's security key.

See "Set a security key for the search head cluster."

The deployer uses the security key to authenticate communication with the cluster members. The cluster members also use it to authenticate with each other. You must set the key to the same value on all cluster members and the deployer. You set the key on the cluster members when you initialize them.

To set the key on the deployer, specify the pass4SymmKey attribute in the [shclustering] stanza of the deployer's server.conf file. For example:

[shclustering]
pass4SymmKey = yoursecuritykey

c. Set the search head cluster label on the deployer.

The search head cluster label is useful for identifying the cluster in the monitoring console. This parameter is optional, but if you configure it on one member, you must configure it with the same value on all members, as well as on the deployer.

To set the label, specify the shcluster_label attribute in the [shclustering] stanza of the deployer's server.conf file. For example:

[shclustering]
shcluster_label = shcluster1

See "Set cluster labels" in Monitoring Splunk Enterprise.

d. Restart the deployer to activate the configuration changes.

3. Install the Splunk Enterprise instances

Install the Splunk Enterprise instances that will serve as cluster members. For information on the minimum number of members necessary, see "Required number of instances."

Caution: Always use new instances. The process of adding an instance to a search head cluster overwrites any configurations or apps currently resident on the instance.

For information on how to install Splunk Enterprise, read the Installation Manual.

Important: You must change the admin password on each instance. The CLI commands that you use to configure the cluster will not operate on instances with the default password.

4. Initialize cluster members

For each instance that you want to include in the cluster, run the splunk init shcluster-config command and restart the instance:

splunk init shcluster-config -auth <username>:<password> -mgmt_uri <URI>:<management_port> -replication_port <replication_port> -replication_factor <n> -conf_deploy_fetch_url <URL>:<management_port> -secret <security_key> -shcluster_label <label>

splunk restart 

Note the following:

• This command is only for cluster members. Do not run this command on the deployer.
• You must use this command to add a search head to a search head cluster. You cannot add a search head through direct editing of .conf files.
• You can only execute this command on an instance that is up and running.
• The -auth parameter specifies your current login credentials for this instance. This parameter is required.
• The -mgmt_uri parameter specifies the URI and management port for this instance. You must use the fully qualified domain name. This parameter is required.
• The -replication_port parameter specifies the port that the instance uses to listen for search artifacts streamed from the other cluster members. You can specify any available, unused port as the replication port. Do not reuse the instance's management or receiving ports. This parameter is required.
• The -replication_factor parameter determines the number of copies of each search artifact that the cluster maintains. All cluster members must use the same replication factor. This parameter is optional. If not explicitly set, the replication factor defaults to 3.
• The -conf_deploy_fetch_url parameter specifies the URL and management port for the deployer instance. This parameter is optional during initialization, but you do need to set it before you can use the deployer functionality. See "Use the deployer to distribute apps and configuration updates."
• The -secret parameter specifies the security key that authenticates communication between the cluster members and between each member and the deployer. The key must be the same across all cluster members and the deployer. See "Set a security key for the search head cluster."
• The -shcluster_label parameter is useful for identifying the cluster in the monitoring console. This parameter is optional, but if you configure it on one member, you must configure it with the same value on all members, as well as on the deployer. See "Set cluster labels" in Monitoring Splunk Enterprise.

For example:

splunk init shcluster-config -auth admin:changed -mgmt_uri https://sh1.example.com:8089 -replication_port 34567 -replication_factor 2 -conf_deploy_fetch_url https://10.160.31.200:8089 -secret mykey -shcluster_label shcluster1

splunk restart 

Caution: To add more members after you bootstrap the captain in step 5, you must follow the procedures in "Add a cluster member".

5. Bring up the cluster captain

a. Select one of the initialized instances to be the first cluster captain. It does not matter which instance you select for this role.

b. Run the splunk bootstrap shcluster-captain command on the selected instance:

splunk bootstrap shcluster-captain -servers_list "<URI>:<management_port>,<URI>:<management_port>,..." -auth <username>:<password>

Note the following:

• This command designates the specified instance as the first cluster captain.
• Run this command on only a single instance.
• The -servers_list parameter contains a comma-separated list of the cluster members, including the member that you are running the command on. The members are identified by URI and management port. This parameter is required.
• Important: The URIs that you specify in -servers_list must be exactly the same as the ones that you specified earlier when you initialized each member, in the -mgmt_uri parameter. You cannot, for example, use https://foo.example.com:8089 during initialization and https://foo.subdomain.example.com:8089 here, even if they resolve to the same node.

Here is an example of the bootstrap command:

splunk bootstrap shcluster-captain -servers_list "https://sh1.example.com:8089,https://sh2.example.com:8089,https://sh3.example.com:8089,https://sh4.example.com:8089" -auth admin:changed

6. Perform post-deployment set-up

To complete set-up, perform these additional steps, as necessary:

a. Connect the search head cluster to search peers. This step is required. It varies according to whether the search peers reside in an indexer cluster:

• To connect the search head cluster to an indexer cluster, see "Integrate the search head cluster with an indexer cluster."

• To connect the search head cluster to non-clustered indexers, see "Connect the search heads in clusters to search peers".

b. Add users. This step is required. See "Add users to the search head cluster".

c. Install a load balancer in front of the search heads. This step is optional. See "Use a load balancer with search head clustering."

d. Use the deployer to distribute apps and configuration updates to the search heads. You must perform this step before you upgrade your set of configurations. See "Use the deployer to distribute apps and configuration updates."

Check search head cluster status

To check the overall status of your search head cluster, run this command from any member:

splunk show shcluster-status -auth <username>:<password>

The command returns basic information on the captain and the cluster members. It indicates the status of each member, such as whether it is up or down.

You can also use the monitoring console to get more information about the status of the cluster. See Use the monitoring console to view search head cluster status and troubleshoot issues.

In addition to checking the status of the search head cluster itself, it is also advisable to check the status of the KV store running on the cluster. Run this command from any member:

splunk show kvstore-status -auth <username>:<password>

See KV store troubleshooting tools.