Back up and restore KV store
Back up the KV store and restore it from backup. Taking regular backups from a healthy environment enables you to restore from a backup in the event of a disaster, or if you add a search head to a cluster. You can also take a backup before migrating to a different machine. See Migrate a Splunk Enterprise instance from one physical machine to another in the Installation Manual for more information.
Make sure to be familiar with the standard backup and restore tools and procedures used by your organization.
Depending on your deployment type, you can perform different tasks with the KV store, including checking the status, taking a backup, and restoring the KV store to an existing or a new search head or search head cluster. Use the following chart to decide which methods to use.
Task | Deployment types | Reasons to choose this task |
---|---|---|
Check the KV store status | Any | Before taking a backup or restoring the KV store, check that that KV store is ready. You can also check on backups and restores that are in progress. |
Backup and restore with point in time consistency | Single instance | This method guarantees consistency. All changes to the KV store during the backup process are captured, and all changes are blocked during the restore process. However, you must ensure that all searches, particularly real-time searches, are complete before restoring the KV store, and you cannot backup specific apps or collection, only the entire KV store. This method is available only for single instance deployments. |
Backup and restore without guaranteed consistency | Any | This method doesn't guarantee consistency. Changes made during backup aren't always captured. This method is available for all types of deployments, and you can choose to backup and restore specific apps or collections, or the entire KV store. |
Check the KV store status
To check the status of the KV store on any deployment type, use the show kvstore-status
command:
./splunk show kvstore-status
The backupRestoreStatus
field and the status
field indicate the statuses of the KV store. The backupRestoreStatus
field indicates the readiness of the node to perform a backup. The status
field indicates the status of the storage engine. Both must be in a ready state in order to take a backup.
Back up and restore the KV store with point in time consistency
Use the following steps to back up the KV store, prepare to restore the KV store data, and then to restore the KV store data. This method is only available for single search head deployments.
Back up the KV store
Complete the following steps to back up the KV store with point in time consistency.
- In the CLI, run the
splunk show kvstore-status
command. - Ensure that the
backupRestoreStatus
field and thestatus
field are both in the ready state. - If you are running any searches that use
outputlookup
with the defaultappend=f
parameter, end them or allow them to complete before taking a backup, or the backup fails. - Optional: Create a separate partition for your backup directory, so that the backup is preserved if the $SPLUNK_DB/kvstore directory fails.
- Use the
splunk backup kvstore -pointInTime true
command from the search head. This creates an archive file in the $SPLUNK_DB/kvstorebackup directory. The-pointInTime true
portion of the command is required to back up with consistency.
To customize your backup, check the full list of arguments for the backup command:
./splunk backup kvstore [-pointInTime <true|false>] [-cancel <true|false>] [-parallelCollections <num>] [-archiveName <archive>]
Argument | Description |
---|---|
-pointInTime | Defaults to false . To take a consistent backup, set it to true .
|
-cancel | Defaults to false . Set the argument to true to cancel an in-progress backup.
|
-parallelCollections | Defaults to 1 . Raise the number to increase the number of collections to back up in parallel.
|
-archiveName | Defaults to kvdump_<epoch>.tar.gz . Set to change the name of the backup file.
|
Prepare to restore the KV store data
Next, complete the following steps to prepare to restore the KV store data:
- Make sure the KV store collection collections.conf file exists on the Splunk instance in the same application name that the KV store is going to be restored to. If you create the collection collections.conf after restoring the KV store data, the KV store data will be lost.
- Ensure that your backup archive file is in the $SPLUNK_DB/kvstorebackup directory.
- Check that you created the backup archive file from the same collection that you are restoring. You cannot restore a backup to a different collection.
You can only restore the KV store with consistency with a backup file that was taken with consistency. Backups taken with consistency used the -pointInTime true
argument in the backup command. Check to see if a backup file was taken with consistency by using the ./splunk show kvstore -archiveName
command.
Restore the KV store data to an existing single search head deployment
Now complete the following steps to restore the KV store data.
Restoring KV store data overwrites any KV store data in your Splunk instance with the data from the backup.
- Ensure all searches are complete, especially real-time searches. To ensure that no searches that use the KV store are started by the scheduler, you can also temporarily disable the scheduler.
- Use the
splunk enable kvstore-maintenance-mode
command to enable maintenance mode. Once you enable maintenance mode, you cannot make any changes to the KV store, and searches that attempt to modify the KV store contents fail. Maintenance mode ensures that the restore completes with consistency. - Restore the KV store data with the
splunk restore kvstore -pointInTime true -archiveName <archive>
command. - Verify that the restore process is complete with the
splunk show kvstore-status
command. - Disable maintenance mode with the
splunk disable kvstore-maintenance-mode
command. If you disabled the scheduler, enable it now.
To customize your restore, check the full list of arguments for the restore command:
./splunk restore kvstore [-pointInTime <true|false>] -archiveName <archive> [-parallelCollection <num>] [-insertionsWorkersPerCollection <num>] [-cancel]
Argument | Description |
---|---|
-pointInTime | Defaults to false . To restore from a backup taken with consistency, set the argument to true .
|
-cancel | Defaults to false . Set the argument to true to cancel an in-progress restore.
|
-parallelCollections | Defaults to 1 . Raise the number to increase the number of collections to restore in parallel, which speeds up the store.
|
-archiveName | Required. Specify the name of the backup file to use. |
-insertionsWorkersPerCollection | Defaults to 1 . Raise the number to increase the number of insertion workers per collection, which speeds up the restore.
|
Back up and restore the KV store without guaranteed consistency
Use the following steps to back up the KV store, prepare to restore the KV store data, and then to restore the KV store data either to an existing deployment or to a new one. This method is available with single search head or clustered deployments.
Back up the KV store
Complete the following steps to back up the KV store:
- In the CLI, run the
splunk show kvstore-status
command. - Ensure that the
backupRestoreStatus
field and thestatus
field are both in theready
state before taking a backup. - Optional: Create a separate partition for your backup directory, so that the backup is preserved if the $SPLUNK_DB/kvstore directory fails.
- Use the
splunk backup kvstore
command from the search head. Alternately, on a search head cluster, run that command from the node with the most recent data. This command creates an archive file in the $SPLUNK_DB/kvstorebackup directory.
Optionally, you can add the following arguments to specify the name of the backup archive file, or specific collections or apps to back up instead of the entire KV store:./splunk backup kvstore [-archiveName <archive>] [-collectionName <collection>] [-appName <app>]
Prepare to restore the KV store data
Complete the following steps to prepare to restore the KV store data.
- Make sure the KV store collection collections.conf file exists on the Splunk instance in the same application name that the KV store is going to be restored to. If you create the collection collections.conf after restoring the KV store data, the KV store data will be lost.
- Ensure that your backup archive file is in the $SPLUNK_DB/kvstorebackup directory. In a clustered environment, ensure it is in this directory on the node from which you are restoring. You only need to restore from one node. The restore replicates across all of the other nodes automatically.
- Check that you created the backup archive file from the same collection that you are restoring. You cannot restore a backup to a different collection.
Restoring KV store data overwrites any KV store data in your Splunk instance with the data from the backup.
Restore the KV store data to an existing search head cluster
Complete the following steps to restore the KV store data to an existing search head cluster:
- Restore the KV store data with the
splunk restore kvstore
command. Optionally, you can add the following arguments to specify the name of the backup archive file, or specific collections or apps to restore instead of the entire KV store:./splunk restore kvstore [-archiveName <archive>] [-collectionName <collection>] [-appName <app>]
- Verify that the restore process is complete by running the
splunk show kvstore-status
command.
Restore the KV store data to a new search head cluster
Complete the following steps to create a new search head cluster with new Splunk Enterprise instances. This procedure only works if you took the backup from a clustered deployment without using the -pointInTime true
argument.
- Back up the KV store data from the same search head in the current search head cluster from which you took the backup.
- On that search head that will be in the new search head cluster environment, create the KV store collection using the same collection name as the KV store data you are restoring.
- Initialize the search head cluster with
replication_factor=1
- Restore the KV store data to the new search head by using the
splunk restore kvstore
command. - Run the following command from the CLI:
splunk clean kvstore --cluster
- Start the Splunk instance and bootstrap with the new search head.
- After the KV store has been restored onto the new search head, add the other new search head cluster members.
- After complete, change the
replication_factor
on each search head to the desired replication factor number. - Perform a rolling restart of your deployment.
Resync the KV store | Migrate the KV store storage engine |
This documentation applies to the following versions of Splunk® Enterprise: 8.2.0, 8.2.1, 8.2.2, 8.2.3, 8.2.4, 8.2.5, 8.2.6, 8.2.7, 8.2.8, 8.2.9, 8.2.10, 8.2.11, 8.2.12
Feedback submitted, thanks!