Migrate the KV store storage engine

Beginning with Splunk Enterprise version 8.1, you can migrate your KV store storage engine from the Memory Mapped (MMAP) storage engine to the WiredTiger storage engine. Migrate your KV store to the WiredTiger storage engine to significantly reduce the amount of storage you need and to improve performance.

Migrating to the WiredTiger storage engine is optional in Splunk Enterprise versions 8.1.* and 8.2.*, but will be mandatory in order to upgrade to Splunk Enterprise versions 9.0 and higher. Plan to migrate your storage engine in the next convenient upgrade window.

Prepare to migrate your storage engine

Complete the following steps to prepare your deployment before you migrate your storage engine.

1. Plan sufficient time for your migration. The time it takes to migrate the KV store storage engine is proportional to the total data in your KV store.
2. In your server.conf file, ensure that enableSplunkdSSL = true. If this setting is false, set it as true and restart Splunk before migrating your storage engine. After your migration is complete, you can choose to return the setting to false or leave it as true.
3. Determine your deployment type. If your single instance of the KV store is located on a search head, the cluster manager, or on any indexer node, you have a single-instance KV store deployment. If you have multiple KV store nodes across a search head cluster, then you have a clustered KV store deployment.
4. Use the following table to choose the migration path that is best for you. The migration depends on whether you use a single-instance KV store deployment or a clustered KV store deployment, and whether you have completed your installation of or upgrade to Splunk Enterprise version 8.1.* or 8.2.*.

Current version	KV store deployment	Instructions
None, new installation of 8.1.* or 8.2.*	Any	Install Splunk Enterprise 8.1 or 8.2.* with the WiredTiger storage engine
8.0 or lower	Single instance	Migrate the KV store during an upgrade to Splunk Enterprise 8.1.* or 8.2.* in a single-instance deployment
8.1.* or 8.2.*	Single instance	Migrate the KV store after an upgrade to Splunk Enterprise 8.1.* or 8.2.* in a single-instance deployment
8.0 or lower	Clustered	Complete the following steps: Upgrade to Splunk Enterprise 8.1 or higher. See How to upgrade Splunk Enterprise in the Installation Manual. Migrate the KV store after an upgrade to Splunk Enterprise 8.1.* or 8.2.* in a clustered deployment.
8.1.* or 8.2.*	Clustered	Migrate the KV store after an upgrade to Splunk Enterprise 8.1.* or 8.2.* in a clustered deployment

Install Splunk Enterprise 8.1.* or 8.2.* with the WiredTiger storage engine

If you are installing Splunk Enterprise 8.1.* or 8.2.* fresh, rather than upgrading from a previous version, complete the following steps to use the WiredTiger storage engine.

1. Download and install Splunk Enterprise version 8.1.* or 8.2.*, but do not start Splunk Enterprise.
2. Open server.conf in the $SPLUNK_HOME/etc/system/local/ directory on *nix or the %SPLUNK_HOME\etc\system\local\ directory on Windows.
3. In the [kvstore] stanza, change the storage engine setting to storageEngine=wiredTiger. If the file does not contain the [kvstore] stanza, paste the following lines into the file.

   [kvstore]
   storageEngine=wiredTiger

   Do not make any other changes to the storage engine settings in the [kvstore] stanza. If the storage engine specified in server.conf does not match the storage engine that Splunk Enterprise is using, then KV store does not start.

4. Save the server.conf file.
5. Start Splunk Enterprise.
6. To check that you are using WiredTiger, use the following command to verify that the status is ready and the storage engine is WiredTiger. In a clustered KV store deployment, use this command on each cluster member:

   splunk show kvstore-status
   

For information about installing Splunk Enterprise, see Installation overview in the Installation Manual.

Migrate the KV store during an upgrade to Splunk Enterprise 8.1.* or 8.2.* in a single-instance deployment

When you are preparing to upgrade your Splunk Enterprise version 8.0 or lower deployment to version 8.1.* or 8.2.* with a single-instance KV store, you can migrate the KV store to the WiredTiger storage engine at the same time. The migration process exports your data to a new directory, restarts the storage engine with WiredTiger, and then restores your exported data.

During the migration process, a backup of the KV store is saved to the $SPLUNK_DB/kvstore/old_db directory on *nix or the %SPLUNK_DB%\kvstore\old_db directory on Windows. You need free storage space equal to twice the size of your KV store directory in order to complete the migration.

1. Open server.conf in the $SPLUNK_HOME/etc/system/local/ directory on *nix or the %SPLUNK_HOME%\etc\system\local\ directory on Windows of Splunk Enterprise.
2. Edit the storageEngineMigration setting to match the following example:

   [kvstore]
   storageEngineMigration=true

3. Save the server.conf file.
4. Begin your upgrade to Splunk Enterprise 8.1 or higher. For information about upgrading Splunk Enterprise, see How to upgrade Splunk Enterprise in the Installation Manual.
5. When prompted to choose whether or not to perform the KV store migration, select yes. If you use the --answer-yes option when you begin to upgrade Splunk Enterprise, then the KV store migration completes automatically during upgrade without prompting.
6. To check that you completed the migration, use the following command to verify that the status is ready and the storage engine is WiredTiger:

   splunk show kvstore-status
   

7. (Optional) After you verify that the migration is successful, delete the KV store backup data in the $SPLUNK_DB/kvstore/old_db directory on *nix or the %SPLUNK_DB%\kvstore\old_db directory on Windows.

Migrate the KV store after an upgrade to Splunk Enterprise 8.1.* or 8.2.* in a single-instance deployment

If you currently use Splunk Enterprise 8.1.* or 8.2.* with a single-instance KV store, you can migrate the KV store to the WiredTiger storage engine without changing what version of Splunk Enterprise that you use. The migration process exports your data to a new directory, restarts the storage engine with WiredTiger enabled, and then restores your exported data.

During the migration process, a backup of the KV store is saved to the $SPLUNK_DB/kvstore/old_db directory on *nix or the %SPLUNK_DB%\kvstore\old_db directory on Windows. You need free storage space equal to twice the size of your KV store directory in order to complete the migration.

1. Stop Splunk Enterprise. Do not use the -f option.
2. Open server.conf in the $SPLUNK_HOME/etc/system/local/ directory.
3. Edit the storageEngineMigration setting to match the following example:

   [kvstore]
   storageEngineMigration=true

4. Save the server.conf file.
5. To begin the migration, use the following command:

   splunk migrate kvstore-storage-engine --target-engine wiredTiger

   To reduce the storage space required to migrate, use the --enable-compression option to compress the backup data. This option causes slightly higher CPU usage.

6. Check the $SPLUNK_HOME/var/log/splunk/mongod.log file on *nix or the %SPLUNK_HOME\var\log\splunk\mongod.log file on Windows to see the status of your migration.
7. After you verify that the migration is successful, start Splunk Enterprise again.
8. (Optional) Delete the KV store backup data in the $SPLUNK_DB/kvstore/old_db directory on *nix or the %SPLUNK_DB%\kvstore\old_db directory on Windows.

Migrate the KV store after an upgrade to Splunk Enterprise 8.1.* or 8.2.* or higher in a clustered deployment

If you currently use Splunk Enterprise 8.1.* or 8.2.* with a clustered KV store, you can migrate the KV store to the WiredTiger storage engine without changing the version of Splunk Enterprise that you are using.

If you are running any searches on a KV store node when you begin migrating, that search might fail. Searches begun after you begin migration are not impacted. Do not do any heavy writes to the KV store while the migration is in process, or the migration can fail.

Use the curl -k -u admin:changeme -X POST https://localhost:8089/services/shcluster/captain/kvmigrate/stop command to stop the migration process at any time.

The KV store non-captain nodes are synced from the captain on a rolling basis, one node at a time, and the migration process does not automatically back up KV store data to a separate location. You can back up your KV store data before you begin the migration process.

Initiate your KV store storage engine migration

Prepare your deployment by editing the server.conf file on each node, and then initiate your migration.

1. Open server.conf in the $SPLUNK_HOME/etc/system/local/ directory on *nix, or the %SPLUNK_HOME\etc\system\local\ directory on Windows.
2. Edit the storageEngineMigration setting to match the following example:

   [kvstore]
   storageEngineMigration=true

3. Complete a rolling restart of your search heads.
4. Check that your instance is ready to migrate by using one of the following commands. You can perform this check with either the REST API or with the Splunk Enterprise command-line interface (CLI).
   REST API:

   curl -k -u admin:changeme https://localhost:8089/services/shcluster/captain/kvmigrate/start -d storageEngine=wiredTiger -d isDryRun=true

   CLI:

   splunk start-shcluster-migration kvstore -storageEngine wiredTiger -isDryRun true

5. Resolve any issues blocking migration. Only perform the migration if all checks pass.
6. When you initiate the migration, you can choose to split up the migration into multiple maintenance windows by migrating based on a percentage of nodes or specific URIs. If you want to migrate specific peers, specify their names and the management port number.
   
   If you specify one of those two options, then you need to initiate and complete the rest of the migration at a later time. If you specify neither option, then all nodes are migrated on a rolling basis one at a time.
   

   Option	REST API sample	CLI sample
   By percentage	curl -k -u admin:changeme https://localhost:8089/services/shcluster/captain/kvmigrate/start -X POST -d storageEngine=wiredTiger -d clusterPerc=50	splunk start-shcluster-migration kvstore -storageEngine wiredTiger -clusterPerc 50
   By URIs	curl -k -u admin:changeme https://localhost:8089/services/shcluster/captain/kvmigrate/start -X POST -d storageEngine=wiredTiger -d peersList="server1:8089,server2:8089,server3:8089"	splunk start-shcluster-migration kvstore -storageEngine wiredTiger -peersList "server1:8089,server2:8089,server3:8089"

Monitor and verify your KV store storage engine migration

Once your migration is in progress, you can use several methods to monitor your migration and verify that it is completed successfully.

• To check which nodes are currently migrating, use the following commands. You can perform this check with either the REST API or with the Splunk Enterprise command-line interface (CLI).
  REST API:

  curl -k -u admin:changeme https://localhost:8089/services/shcluster/captain/kvmigrate/status

  CLI:

  splunk show shcluster-kvmigration-status

• For more information about the status of the upgrade, use the following command:

  splunk show kvstore-status

• To check the progress of the migration of a cluster member, see the KVStoreReplicaSetStats entry in the $SPLUNK_HOME/var/log/introspection/kvstore.log file on *nix, or the %SPLUNK_HOME\var\log\introspection\kvstore.log file on Windows, on that member. This status updates every 30 seconds.

If you backed up your KV store, verify that the migration is successful and then delete the KV store backup data.