Back up and restore search head cluster settings

Search head clusters can usually recover from member failures without the need to manually restore configuration settings. Several of the other topics in this chapter provide guidance on recovering from various sorts of member failure, In particular, see:

• Handle failure of a cluster member
• Use static captain to recover from loss of majority

In a functioning search head cluster, each member continually replicates changes to its state to the other members. This makes it possible to rebuild your cluster even if only one member remains intact.

However, to deal with catastrophic failure of a search head cluster, such as the failure of a data center, you can periodically back up the cluster state, so that you can later restore that state to a new or standby cluster, if necessary.

In addition, to deal with failure of the deployer, you can backup and restore the deployer's configuration bundle.

As with any backup-and-recovery scheme, test that these procedures work for you before you need them to work for you.

Backup the search head cluster settings

A backup of all search head cluster configurations requires two backups:

• The search head cluster state
• The deployer's configuration bundle

Backup the search head cluster state

On a cluster member, preferably the current captain:

1. Back up the most recent set of replicated configurations, located at $SPLUNK_HOME/var/run/splunk/snapshot/$LATEST_TIME-$CHECKSUM.bundle.
2. Back up the $SPLUNK_HOME/etc/system/local/server.conf file.
   Note: The only setting from this file that you will use when restoring from this backup is the id setting under the [shclustering] stanza. This setting is a unique identifier for the cluster, shared by all cluster members.
3. Back up the KV store:

   splunk backup kvstore
   

   This command creates an archive file in the $SPLUNK_HOME/var/lib/splunk/kvstorebackup directory. See Back up the KV store in the Admin Manual.

4. Create a tarball containing the set of backups. This is your search head cluster configuration backup. Store it somewhere safe.

Backup the deployer's configuration bundle

Back up the deployer's $SPLUNK_HOME/etc/shcluster directory. This directory contains the configuration bundle that gets deployed to all cluster members.

Restore the search head cluster settings

You can restore the settings to either a new or an existing, standby cluster. The procedure documented here assumes that you are restoring to a standby cluster, but you can apply the main points of the procedure to a new cluster.

To restore a cluster's settings, restore two sets of configurations:

• The deployer's configuration bundle
• The search head cluster state

All members of both the old and new clusters, along with their deployers, must be running the same version of Splunk Enterprise, down to the maintenance level.

Restore the deployer's configuration bundle

This procedure assumes that you are restoring to a new deployer. If the old deployer is intact, you can reuse it by just pointing the new cluster members to it.

A deployer can only service a single cluster. The old cluster must be permanently inactive before you can use the existing deployer with the new cluster.

1. Stop all members of the standby search head cluster.
2. Copy the backup of the configuration bundle to the new deployer's $SPLUNK_HOME/etc/shcluster directory, overwriting the existing contents, if any.
3. Run the splunk apply shcluster-bundle command on the deployer:

   splunk apply shcluster-bundle -answer-yes -target <URI>:<management_port> -auth <username>:<password>
   

   See Push the configuration bundle.

Do not restart the standby cluster members at this point.

Restore the search head cluster state

1. Confirm that all members of the standby search head cluster are still stopped.
2. Untar the set of backups to a temporary location.
3. On each standby cluster member:
   1. Restore the replicated configurations:
      1. Move the replicated bundle $LATEST_TIME-$CHECKSUM.bundle from the temporary location to $SPLUNK_HOME/etc.
      2. Untar $LATEST_TIME-$CHECKSUM.bundle.

         You must be working in the $SPLUNK_HOME/etc directory when you untar $LATEST_TIME-$CHECKSUM.bundle. The files in $LATEST_TIME-$CHECKSUM.bundle are relative to $SPLUNK_HOME/etc.

      3. To confirm that the files untarred properly, check for the presence of files in their proper location; for example, look for $SPLUNK_HOME/etc/system/replication/ops.json.
   2. Restore the KV store configurations. Follow the instructions in Restore the KV store data in the Admin Manual.
   3. Restore the search head cluster id field. Edit $SPLUNK_HOME/etc/system/local/server.conf and change the id setting in the shclustering stanza to use the value from the backup.
4. Start all cluster members.
5. Wait a few minutes for captain election to complete and for the deployer configuration bundle to be applied.