Splunk® Enterprise

Admin Manual

Acrobat logo Download manual as PDF

Acrobat logo Download topic as PDF

Migrate the KV store storage engine

Beginning with Splunk Enterprise version 8.1, you can migrate your KV store storage engine from the Memory Mapped (MMAP) storage engine to the WiredTiger storage engine. Migrate your KV store to the WiredTiger storage engine to significantly reduce the amount of storage you need and to improve performance.

Use the following table to choose the migration path that is best for you. The migration depends on whether you have completed your installation of or upgrade to Splunk Enterprise version 8.1, and whether you use a single-instance deployment or a clustered deployment.

If your single instance of the KV store is located on a search head, the cluster manager, or on any indexer node, complete the steps for a single-instance KV store deployment. If you have multiple KV store nodes across a search head cluster, then complete the steps for a clustered KV store deployment.

The time it takes to migrate the KV store storage engine is proportional to the total data in your KV store.

Current version KV store deployment Instructions
None, new installation of 8.1 Any Install Splunk Enterprise 8.1 with the WiredTiger storage engine
8.0 or earlier Single instance Migrate KV store during an upgrade to Splunk Enterprise 8.1 in a single-instance deployment
8.1 or later Single instance Migrate KV store after an upgrade to Splunk Enterprise 8.1 in a single-instance deployment
8.0 or earlier Clustered Complete the following steps:
  1. Upgrade to Splunk Enterprise 8.1. See How to upgrade Splunk Enterprise in the Installation Manual.
  2. Migrate KV store after an upgrade to Splunk Enterprise 8.1 in a clustered deployment.
8.1 or later Clustered Migrate KV store after an upgrade to Splunk Enterprise 8.1 in a clustered deployment

Install Splunk Enterprise 8.1 with the WiredTiger storage engine

If you are installing Splunk Enterprise 8.1 fresh, rather than upgrading from a previous version, complete the following steps to use the WiredTiger storage engine.

  1. Download and install Splunk Enterprise version 8.1, but do not start Splunk Enterprise.
  2. Open server.conf in the $SPLUNK_HOME/etc/system/local/ directory on *nix or the %SPLUNK_HOME\etc\system\local\ directory on Windows.
  3. In the [kvstore] stanza, change the storage engine setting to storageEngine=wiredTiger. If the file does not contain the [kvstore] stanza, paste the following lines into the file.
    [kvstore]
    storageEngine=wiredTiger

    Do not make any other changes to the storage engine settings in the [kvstore] stanza. If the storage engine specified in server.conf does not match the storage engine that Splunk Enterprise is using, then KV store does not start.

  4. Save the server.conf file.
  5. Start Splunk Enterprise.
  6. To check that you are using WiredTiger, use the following command to verify that the status is ready and the storage engine is WiredTiger. In a clustered KV store deployment, use this command on each cluster member:
    splunk show kvstore-status
    

For information about installing Splunk Enterprise, see Installation overview in the Installation Manual.

Migrate the KV store during an upgrade to Splunk Enterprise 8.1 in a single-instance deployment

When you are preparing to upgrade your Splunk Enterprise version 8.0 or lower deployment to version 8.1 with a single-instance KV store, you can migrate the KV store to the WiredTiger storage engine at the same time. The migration process exports your data to a new directory, restarts the storage engine with WiredTiger, and then restores your exported data.

During the migration process, a backup of the KV store is saved to the $SPLUNK_DB/kvstore/old_db directory on *nix or the %SPLUNK_DB%\kvstore\old_db directory on Windows. You need free storage space equal to twice the size of your KV store directory in order to complete the migration.

  1. Open server.conf in the $SPLUNK_HOME/etc/system/local/ directory on *nix or the %SPLUNK_HOME%\etc\system\local\ directory on Windows of Splunk Enterprise 8.1.
  2. Edit the storageEngineMigration setting to match the following example:
    [kvstore]
    storageEngineMigration=true
  3. Save the server.conf file.
  4. Begin your upgrade to Splunk Enterprise 8.1. For information about upgrading to Splunk Enterprise 8.1, see How to upgrade Splunk Enterprise in the Installation Manual.
  5. When prompted to choose whether or not to perform the KV store migration, select yes. If you use the --answer-yes option when you begin to upgrade Splunk Enterprise, then the KV store migration completes automatically during upgrade without prompting.
  6. To check that you completed the migration, use the following command to verify that the status is ready and the storage engine is WiredTiger:
    splunk show kvstore-status
    
  7. (Optional) After you verify that the migration is successful, delete the KV store backup data in the $SPLUNK_DB/kvstore/old_db directory on *nix or the %SPLUNK_DB%\kvstore\old_db directory on Windows.


Migrate the KV store after an upgrade to Splunk Enterprise 8.1 in a single-instance deployment

If you currently use Splunk Enterprise 8.1 with a single-instance KV store, you can migrate the KV store to the WiredTiger storage engine. The migration process exports your data to a new directory, restarts the storage engine with WiredTiger enabled, and then restores your exported data.

During the migration process, a backup of the KV store is saved to the $SPLUNK_DB/kvstore/old_db directory on *nix or the %SPLUNK_DB%\kvstore\old_db directory on Windows. You need free storage space equal to twice the size of your KV store directory in order to complete the migration.

  1. Stop Splunk Enterprise. Do not use the -f option.
  2. Open server.conf in the $SPLUNK_HOME/etc/system/local/ directory.
  3. Edit the storageEngineMigration setting to match the following example:
    [kvstore]
    storageEngineMigration=true
  4. Save the server.conf file.
  5. To begin the migration, use the following command:
    splunk migrate kvstore-storage-engine --target-engine wiredTiger

    To reduce the storage space required to migrate, use the --enable-compression option to compress the backup data. This option causes slightly higher CPU usage.

  6. Check the $SPLUNK_HOME/var/log/splunk/mongod.log file on *nix or the %SPLUNK_HOME\var\log\splunk\mongod.log file on Windows to see the status of your migration.
  7. After you verify that the migration is successful, start Splunk Enterprise again.
  8. (Optional) Delete the KV store backup data in the $SPLUNK_DB/kvstore/old_db directory on *nix or the %SPLUNK_DB%\kvstore\old_db directory on Windows.

Migrate the KV store after an upgrade to Splunk Enterprise 8.1 in a clustered deployment

If you currently use Splunk Enterprise 8.1 with a clustered KV store, you can migrate the KV store to the WiredTiger storage engine.

If you are running any searches on a KV store node when you begin migrating, that search might fail. Searches begun after you begin migration are not impacted. Do not do any heavy writes to the KV store while the migration is in process, or the migration can fail.

Use the curl -k -u admin:changeme https://localhost:8099/services/shcluster/captain/kvmigrate/stop command to stop the migration process at any time.

The KV store non-captain nodes are synced from the captain on a rolling basis, one node at a time, and the migration process does not automatically back up KV store data to a separate location. You can back up your KV store data before you begin the migration process.

Initiate your KV store storage engine migration

Prepare your deployment by editing the server.conf file on each node, and then initiate your migration.

  1. Open server.conf in the $SPLUNK_HOME/etc/system/local/ directory on *nix, or the %SPLUNK_HOME\etc\system\local\ directory on Windows.
  2. Edit the storageEngineMigration setting to match the following example:
    [kvstore]
    storageEngineMigration=true
  3. Restart the search head, and repeat these steps for each search head.
  4. Check that your instance is ready to migrate by using one of the following commands. You can perform this check with either the REST API or with the Splunk Enterprise command-line interface (CLI).
    REST API:
    curl -k -u admin:changeme https://localhost:8099/services/shcluster/captain/kvmigrate/start -d storageEngine=wiredTiger -d isDryRun=true
    CLI:
    splunk start-shcluster-migration kvstore -storageEngine wiredTiger -isDryRun
  5. Resolve any issues blocking migration. Only perform the migration if all checks pass.
  6. To initiate the migration, choose if you want to migrate based on a percentage of nodes, or based on specific URIs. If you specify neither option, then all nodes are migrated on a rolling basis one at a time.
    Option REST API sample CLI sample
    By percentage
    curl -k -u admin:changeme https://localhost:8099/services/shcluster/captain/kvmigrate/start -X POST 
    -d storageEngine=wiredTiger 
    -d clusterPerc=50
    splunk start-shcluster-migration kvstore 
    -storageEngine wiredTiger 
    -clusterPerc 50
    By URIs
    curl -k -u admin:changeme https://localhost:8099/services/shcluster/captain/kvmigrate/start -X POST 
    -d storageEngine=wiredTiger 
    -d peersList="server1:8192,server2:8192,server3:8192"
    splunk start-shcluster-migration kvstore 
    -storageEngine wiredTiger  
    -peersList "server1:8192,server2:8192,server3:8192"

Monitor and verify your KV store storage engine migration

Once your migration is in progress, you can use several methods to monitor your migration and verify that it is completed successfully.

  • To check which nodes are currently migrating, use the following commands. You can perform this check with either the REST API or with the Splunk Enterprise command-line interface (CLI).
    REST API:
    curl -k -u admin:changeme https://localhost:8099/services/shcluster/captain/kvmigrate/status
    CLI:
    splunk show shcluster-kvmigration-status
  • For more information about the status of the upgrade, use the following command:
    splunk show kvstore-status
  • To check the progress of the migration of a cluster member, see the KVStoreReplicaSetStats entry in the $SPLUNK_HOME/var/log/introspection/kvstore.log file on *nix, or the %SPLUNK_HOME\var\log\introspection\kvstore.log file on Windows, on that member. This status updates every 30 seconds.

If you backed up your KV store, verify that the migration is successful and then delete the KV store backup data.

Last modified on 29 October, 2020
PREVIOUS
Back up and restore KV store
  NEXT
KV store troubleshooting tools

This documentation applies to the following versions of Splunk® Enterprise: 8.1.0


Was this documentation topic helpful?

You must be logged into splunk.com in order to post comments. Log in now.

Please try to keep this discussion focused on the content covered in this documentation topic. If you have a more general question about Splunk functionality or are experiencing a difficulty with Splunk, consider posting a question to Splunkbase Answers.

0 out of 1000 Characters