Configure the search head cluster

This topic describes how to configure the behavior of the search head cluster itself. It does not describe how to configure the search-time environment of the cluster members, such as the set of saved searches, dashboards, and apps that the members have access to. For information on configuring the search-time environment, see the chapter "Update search head cluster members".

The members store their cluster configurations in their local server.conf files, located under $SPLUNK_HOME/etc/system/local/. See the server.conf specification file for details on all available configuration attributes.

Key information

Remember these key points while reading this topic:

• The essential configuration occurs when you initialize each member during the deployment process.
• Search head clustering has a large number of configuration settings available. With a few exceptions, you should not change these settings from their initial or default values without guidance from Splunk Support.
• You must maintain identical settings across all members, except as noted.
• When you do change a setting across all members, you must restart all the members at approximately the same time.

Initialization-time configurations

You can set all essential configurations during the deployment process, when you initialize each member. These are the key configuration attributes that you can or must set for each cluster member during initialization:

• The member's URI. See "Deploy a search head cluster".
• The member's replication port. See "Deploy a search head cluster".
• The cluster's replication factor. See "Choose the replication factor for the search head cluster".
• The cluster's security key. See "Set a security key for the search head cluster".
• The deployer location. See "Point the cluster members to the deployer".
• The cluster's label. See "Deploy a search head cluster".

Caution: It is strongly recommended that you set all these attributes during initialization and do not later change them. See "Deploy a search head cluster".

Post-initialization configuration changes

The main configuration changes that you can safely perform on your own, post-initialization, are the ad hoc search settings. There are two of these: one for specifying whether a particular member should run ad hoc searches only, and another for specifying whether the member currently functioning as the captain should run ad hoc searches only. The captain will not assign scheduled searches to ad hoc members. See "Configure a cluster member to run ad hoc searches only".

You can also temporarily switch to a static captain, as a work around for disaster recovery. See "Use static captain to recover from loss of majority."

Caution: Do not edit the id attribute in the [shclustering] stanza. The system sets it automatically. This attribute must conform to the requirements for a valid GUID.

Set the search head cluster label

You usually set the cluster label with the splunk init command when you deploy the cluster. If you did not set it during deployment, you can later set it for the cluster by running this command on any one member:

splunk edit shcluster-config -shcluster_label <label>

You do not need to restart the member after setting the label.

Note: If you set the label on a cluster member, you must also set it on the deployer. See "Configure the deployer."

The -shcluster_label parameter is useful for identifying the cluster in the monitoring console. See "Set cluster labels" in Monitoring Splunk Enterprise.

Maintain the same configuration settings across all members

The server.conf attributes for search head clustering must have the same values across all members, with these exceptions:

• mgmt_uri
• adhoc_searchhead
• [replication_port://<port>]

If any configuration values other than these ones vary from member to member, then the behavior of the cluster will change depending on which member is currently serving as captain. You do not want that to occur.

Configuration methods

Most of the configuration occurs during initial cluster deployment, through the CLI splunk init command. To perform further configuration later, you have two choices:

• Use the CLI splunk edit shcluster-config command.

• Edit the [shclustering] stanza in server.conf directly.

It is generally simpler to use the CLI.

Caution: You must make the same configuration changes on all members and then restart them all at approximately the same time. Because of the importance of maintaining identical settings across all members, do not use the splunk rolling-restart command to restart, except when changing the captain_is_adhoc_searchhead attribute, as described in "Configure a cluster member to run ad hoc searches only". Instead, run the splunk restart command on each member.

Configure search head clustering with the CLI

You can use the CLI splunk edit shcluster-config command to make edits to the [shclustering] stanza in server.conf. Specify each attribute and its configured value as a key value pair.

For example, to edit the adhoc_searchhead attribute:

splunk edit shcluster-config -adhoc_searchhead true -auth <username>:<password>

The CLI confirms that the operation was successful and instructs you to restart splunkd.

Note the following:

• You can use this command to edit any attribute in the [shclustering] stanza except the disabled attribute, which turns search head clustering on and off.
• You can only use this command on a member that has already been initialized. For initial configuration, use splunk init shcluster-config.

Configure search head clustering by editing server.conf

You can also change attributes by directly editing server.conf. The search head clustering attributes are located in the [shclustering] stanza, with one exception: To modify the replication port, use the [replication_port] stanza.