Splunk® Enterprise

Managing Indexers and Clusters of Indexers

Download manual as PDF

Download topic as PDF

Replace the master node on the indexer cluster

You might need to replace the master node for either of these reasons:

  • The node fails.
  • You must move the master to a different machine or site.

Although there is currently no master failover capability, you can prepare the indexer cluster for master failure by configuring a stand-by master that you can immediately bring up if the primary master goes down. You can use the same method to replace the master intentionally.

This topic describes the key steps in replacing the master:

  1. Back up the files that the replacement master needs.

    This is a preparatory step. You must do this before the master fails or otherwise leaves the system.

  2. Ensure that the peer and search head nodes can find the new master.
  3. Replace the master.

In the case of a multisite cluster, you must also prepare for the possible failure of the site that houses the master. See Handle master site failure.

Back up the files that the replacement master needs

There are two static configurations that you must backup so that you can later copy them to the replacement master:

  • The master's server.conf file, which is where the master cluster settings are stored. You must back up this file whenever you change the master's cluster configuration.
  • The master's $SPLUNK_HOME/etc/master-apps directory, which is where common peer configurations are stored, as described in Update cluster peer configurations. You must back up this directory whenever you update the set of content that you push to the peer nodes.

In addition to the above configuration files, make sure to back up any other configuration files that you have customized on cluster master, such as inputs.conf, web.conf, and so on.

In preparing a replacement master, you must copy over only the master's static state. You do not copy or otherwise deal with the dynamic state of the cluster. The cluster peers as a group hold all information about the dynamic state of a cluster, such as the status of all bucket copies. They communicate this information to the master node as necessary, for example, when a downed master returns to the cluster or when a stand-by master replaces a downed master. The master then uses that information to rebuild its map of the cluster's dynamic state.

Ensure that the peer and search head nodes can find the new master

You can choose between two approaches for ensuring that the peer nodes and search head can locate the replacement instance and recognize it as the master:

  • The replacement uses the same IP address and management port as the primary master. To ensure that the replacement uses the same IP address, you must employ DNS-based failover, a load balancer, or some other technique. The management port is set during installation, but you can change it by editing web.conf.
  • The replacement does not use the same IP address or management port as the primary master. In this case, after you bring up the new master, you must update the master_uri setting on all the peers and search heads to point to the new master's IP address and management port.

Neither approach requires a restart of the peer or search head nodes.

Replace the master

Prerequisite

You must have up-to-date backups of the two sets of static configuration files, as described in Back up the files that the replacement master needs.

Steps

If you want to skip steps 3 and 5, you can replace the [general] and [clustering] stanzas on the replacement master in step 4, instead of copying the entire server.conf file.

  1. Stop the old master, if this is a planned replacement. If the replacement is due to a failed master, then this step has already been accomplished for you.
  2. Install, start, and stop a new Splunk Enterprise instance. Alternatively, you can reuse an existing instance that is not needed for another purpose. This will be the replacement master.
  3. Copy the sslPassword setting from the replacement master's server.conf file to a temporary location.

    In release 6.5, the sslKeysfilePassword attribute was deprecated and replaced by the sslPassword attribute. If the server.conf file is using sslKeysfilePassword, then copy that setting instead.

  4. Copy the backup of the old master's server.conf and $SPLUNK_HOME/etc/master-apps files to the replacement master.
  5. Delete the sslPassword setting in the copied server.conf, and replace it with the version of the setting that you saved in step 3.
  6. Start the replacement master.
  7. Make sure that the peer and search head nodes are pointing to the new master through one of the methods described in Ensure that the peer and search head nodes can find the new master.

For information on the consequences of a master failing, see What happens when the master node goes down.

PREVIOUS
Configure the master with the CLI
  NEXT
Peer node configuration overview

This documentation applies to the following versions of Splunk® Enterprise: 6.5.0, 6.5.1, 6.5.1612 (Splunk Cloud only), 6.5.2, 6.5.3, 6.5.4, 6.5.5, 6.5.6, 6.5.7, 6.5.8, 6.5.9, 6.5.10, 6.6.0, 6.6.1, 6.6.2, 6.6.3, 6.6.4, 6.6.5, 6.6.6, 6.6.7, 6.6.8, 6.6.9, 6.6.10, 6.6.11, 6.6.12, 7.0.0, 7.0.1, 7.0.2, 7.0.3, 7.0.4, 7.0.5, 7.0.6, 7.0.7, 7.0.8, 7.0.9, 7.0.10, 7.0.11, 7.1.0, 7.1.1, 7.1.2, 7.1.3, 7.1.4, 7.1.5, 7.1.6, 7.1.7, 7.1.8, 7.2.0, 7.2.1, 7.2.2, 7.2.3, 7.2.4, 7.2.5, 7.2.6, 7.2.7, 7.2.8, 7.3.0, 7.3.1


Comments

Hi,

under "Ensure that the peer and search head nodes can find the new master" there is stated that "Neither approach requires a restart of the peer or search head nodes.".

Is this really correct?
In my opinion changing master_uri setting in server.conf requires a restart.

Can you please clarify this?

Thanks!
- Lorenz

DATEVeG
August 9, 2019

Was this documentation topic helpful?

Enter your email address, and someone from the documentation team will respond to you:

Please provide your comments here. Ask a question or make a suggestion.

You must be logged into splunk.com in order to post comments. Log in now.

Please try to keep this discussion focused on the content covered in this documentation topic. If you have a more general question about Splunk functionality or are experiencing a difficulty with Splunk, consider posting a question to Splunkbase Answers.

0 out of 1000 Characters