Back up KV Store

This topic describes how to safely back up and restore your KV Store.

Back up the KV Store

Before performing these steps make sure to be familiar with the standard backup and restore tools and procedures used by your organization.

1. To back up KV store data, first shut down the Splunk instance from which the KV Store will be backed up.
2. Back up all files in the path that is specified in the dbPath parameter of the [kvstore] stanza in the server.conf file.
3. On a single node, back up the kvstore folder found in your $SPLUNK_DB path. By default the path is /var/lib/splunk/kvstore.

If using a search head cluster, back up the KV Store data on any cluster member.

For general information about backup strategies in Splunk Enterprise, see Choose your backup strategy in the Managing Indexers and Clusters of Indexers manual.

Restore the KV Store data

Note: In order to successfully restore KV Store data, the KV Store collection collections.conf must already exist on the Splunk instance the KV Store will be restored to. If you create the collection collections.conf after restoring the KV Store data, then the KV Store data will be lost.

To restore the KV Store data to the same search head cluster that it was backed up from, restore the kvstore folder on each cluster member. For example, in a three-member search head cluster:

1. Back up the KV Store data from a member of the search head cluster.
2. Stop each cluster member.
3. Restore the backed-up KV Store data folder to each cluster member.
4. Start each cluster member.

Restore the KV Store data to a new member being added to the search head cluster

Restore the KV Store data to the new member and add the new member to the cluster. For example, in a three-member search head cluster:

1. Back up the KV Store data from a member of the search head cluster.
2. On the search head that you want to add to the search head cluster:
   1. Add the member to the cluster. See "Add a cluster member" in the Distributed Search manual.
   2. Stop the member.
   3. Restore the KV Store data.
   4. Start the new member.

Restore the KV Store data from an old search head cluster to a new search head cluster

Note: This procedure assumes you are creating a new search head cluster with new Splunk Enterprise instances.

When more than half of the KV store members are unavailable or are stale, you need to recreate the search head cluster and restore the KV store data.

Decommission the old KV store

Before you shut down the old KV store, back up the data.

1. Back up the KV store data in the old search head cluster.
2. Stop all search head instances.
3. Clear the KV store data using the command splunk clean kvstore --cluster.
4. Back up the data in the [shclustering] stanza of the server.conf file.

Restore KV store data to the new search head cluster

1. Create a new search head cluster without bootstrapping the cluster. In the new cluster, name the new KV store collection with the same collection name as the KV store data you are restoring.
2. Choose a member in the new cluster and initialize it with replication_factor=1 in the [shclustering] stanza of the server.conf file. See Deploy a search head cluster in Distributed Search.
3. Stop the new member, then restore the KV store data to that instance.
4. Restart the instance. Bootstrap the cluster with only the new member.
5. Verify that the search head cluster status and the KV store status are operational. Verify that the KV store data exists and is accurate.
6. Stop the new search head cluster instances. Restore the old [shclustering] stanza data in the new server.conf files for all instances, but leave the default values for the mgmt_uri and id variables.
7. Clean the folder $SPLUNK_HOME/var/run/splunk/_raft/ on the instance used to bootstrap the cluster.
8. Start all search head members and bootstrap the cluster with all members.
9. Verify that the search head cluster status and the KV store status are operational.