Managing Indexers and Clusters of Indexers

Configure the master

You configure the master when you enable it, as described in "Enable the master node". This is usually all the configuration the master needs.

Perform post-deployment configuration

If you do need to perform further configuration, you have these choices:

• You can edit the configuration from the master node dashboard in Splunk Web, as described in "View the master dashboard".

• You can directly edit the [clustering] stanza in the master's underlying server.conf file. See "Configure cluster components with server.conf" for details. Some advanced settings can only be configured by directly editing this file.

• You can use the CLI edit cluster-config command. See "Configure the cluster with the CLI" for details.

Warning: Although it is possible to change the settings for the replication factor and search factor, it is inadvisable to increase either of them once your cluster contains significant amounts of data. Doing so will kick off a great deal of bucket activity, which will have an adverse effect on the cluster's performance while bucket copies are being created and/or made searchable.

You should not change the heartbeat_timeout from its default value of 60 (seconds) unless instructed to do so by Splunk Support. In particular, do not decrease it, as doing so could lead to problems if your peers become overloaded.

After you change the master configuration, you must restart the master for the changes to take effect.

Important: The master has the sole function of managing the other cluster components. You cannot use it to index external data or to search the cluster.

Configure a stand-by master

Although there is currently no master failover capability, you can prepare for master failures by configuring a stand-by master that you can immediately bring up if the primary master goes down for any reason.

Back up the files that the stand-by master needs

In preparing a stand-by master, you only need to copy over the master's static state. The cluster peers as a group hold all information about the dynamic state of a cluster, such as the status of all cluster copies. They will communicate this information to the master node as necessary; for example, when a downed master returns to the cluster or - barring that - when a stand-by master replaces a downed master. The master will then use that information to rebuild its map of the cluster's state.

There are two separate static configurations on the master that you need to back up so that you can later copy them to the stand-by master:

• The master's server.conf file, which is where the master cluster settings are stored. You need to back up this file whenever you change the master's cluster configuration.

• The master's $SPLUNK_HOME/etc/master-apps directory, which is where common peer configurations are stored, as described in "Update common peer configurations". You need to back up this directory whenever you update the set of content you're pushing to the peer nodes.

Ensure that the peer and search head nodes can find the new master

You can choose between two approaches for ensuring that the peer nodes and search head can locate the stand-by instance and recognize it as the master:

• The stand-by uses the same IP address and management port as the primary master. The management port is set during installation but can be changed later by editing web.conf. To ensure that the stand-by uses the same IP address, you need to employ DNS-based failover, a load balancer, or some other technique.

• The stand-by does not use the same IP address or management port as the primary master. In this case, after you bring up the new master, you must update the master_uri setting on all the peers and search head to point to the new master's IP address and management port.

Neither approach requires a restart of the peer or search head nodes.

Replace the master

Assuming that you have backed up the two sets of static configuration files, it's a simple process to replace a downed master with the stand-by:

1. Install, start, and stop a new Splunk Enterprise instance. This will be the stand-by master.

2. Copy the sslKeysfilePassword setting from the stand-by master's server.conf file to a temporary location.

3. Copy the back-up of the old master's server.conf and $SPLUNK_HOME/etc/master-apps files to the stand-by master.

4. Delete the sslKeysfilePassword setting in the copied server.conf and replace it with the version of the setting saved in step 2.

5. Start the stand-by master.

6. Make sure that the peer and search head nodes are pointing to the new master through one of the methods described above in "Ensure that the peer and search head nodes can find the new master".

Note: If you want to skip steps 2 and 4, you can simply replace the [general] and [clustering] stanzas on the stand-by master instead of copying the entire server.conf file.

For information on the consequences of a master failing, see "What happens when a master node goes down".