Configure the indexer cluster with server.conf

Before reading this topic, see "About configuration files" and the topics that follow it in the Admin Manual. Those topics explain how Splunk Enterprise uses configuration files.

Indexer cluster settings reside in the server.conf file, located in $SPLUNK_HOME/etc/system/local/. When you deploy a cluster node through Splunk Web or the CLI, the node saves the settings to that file. You can also edit server.conf file directly, either to deploy initially or to change settings later.

The main server.conf stanza that controls indexer clustering is [clustering]. Besides the basic attributes that correspond to settings in Splunk Web, server.conf provides a number of advanced settings that control communication between cluster nodes. Unless advised by Splunk Support, do not change those settings.

This topic discusses some issues that are common to all node types.

Configure the various node types

For specific instructions for each node type, see:

• "Configure the manager node with server.conf".
• "Configure peer nodes with server.conf".
• "Configure the search head with server.conf".

For details on all the clustering attributes, including the advanced ones, read the server.conf specification.

For multisite cluster configurations, also read "Configure multisite indexer clusters with server.conf".

Configure the security key

Set the pass4SymmKey attribute to configure a security key that authenticates communication between the manager node, peers, and search heads. You must use the same key value for all cluster nodes.

You set pass4SymmKey when you deploy the cluster. For details on how to set the key on the manager node, see Enable the indexer cluster manager node. You also set it when when enabling the peer nodes and search heads.

If you set the key directly in server.conf, you must set it inside the [clustering] stanza for indexer clustering.

Important: Save a copy of the key in a safe place. Once an instance starts running, the security key changes from clear text to encrypted form, and it is no longer recoverable from server.conf. If you later want to add a new node, you will need to use the clear text version to set the key.

For information on setting the security key for a combined search head cluster and indexer cluster, see Integrate the search head cluster with an indexer cluster in Distributed Search.

Restart after modifying server.conf?

After you configure an instance as a cluster node for the first time, you need to restart it for the change to take effect.

If you make a configuration change later on, you might not need to restart the instance, depending on the type of change. Avoid restarting peers when possible. Restarting the set of peers can result in prolonged amounts of bucket-fixing.

Initial configuration

After initially configuring instances as cluster nodes, you need to restart all of them (manager node, peers, and search head) for the changes to take effect. You can do this by invoking the CLI restart command on each node:

$SPLUNK_HOME/bin/splunk restart

When the manager node starts up for the first time, it blocks indexing on the peers until you enable and restart the replication factor number of peers. Do not restart the manager while it is waiting for the peers to join the cluster. If you do, you will need to restart the peers a second time.

Important: Although you can use the CLI restart command when you initially enable an instance as a cluster peer node, do not use it for subsequent restarts. The restart command is not compatible with index replication once replication has begun. For more information, including a discussion of safe restart methods, read "Restart a single peer".

Subsequent configuration changes

If you change any of the following attributes in the server.conf file, you do not need to restart the node.

On a peer node:

• manager_uri
• notify_scan_period

On a search head:

• manager_uri

On a manager node:

• quiet_period
• heartbeat_timeout
• restart_timeout
• max_peer_build_load
• max_peer_rep_load
• cluster_label
• access_logging_for_heartbeats
• use_batch_mask_changes
• percent_peers_to_restart
• summary_replication

All other cluster-related configuration changes require a restart.