Migrate the KV store storage engine

Splunk Enterprise versions 9.0 and higher require the WiredTiger storage engine and server version 4.2, which significantly reduces the amount of storage you need and improves performance. Migrate to WiredTiger either before or during upgrade to Splunk Enterprise 9.0, and then upgrade to server version 4.2. Migrating your storage engine before or during upgrade to Splunk Enterprise 9.0 or higher is a best practice, but migrating immediately after upgrade is required.

All Splunk Enterprise versions 8.1 and higher support WiredTiger, so you can consider migrating to WiredTiger before your upgrade to reduce downtime during the upgrade. If you prefer to perform the WiredTiger migration and the upgrade to Splunk Enterprise at separate times, check the documentation for your current version of Splunk Enterprise to complete your migration before initiating your upgrade to Splunk Enterprise 9.0 or higher.

To migrate your KV store storage engine during your upgrade to Splunk Enterprise 9.0 or higher, first determine your deployment type. If your single instance of the KV store is located on a search head, the cluster manager, or any indexer node, you have a single-instance KV store deployment. If you have multiple KV store nodes across a search head cluster, then you have a clustered KV store deployment. Complete the steps associated with your deployment type.

After completing your migration to WiredTiger and server version 4.2, you can optionally remove the unsupported binary files for previous versions of MongoDB.

Migrate the KV store in a single-instance deployment

Single-instance deployments of Splunk Enterprise 9.0 and higher are automatically migrated to the WiredTiger storage engine and the latest version of MongoDB, server version 4.2, during the upgrade.

1. Complete any prompts during your Splunk Enterprise upgrade.
2. Verify that you have the latest version of the storage engine after upgrade with the following command:

   splunk show kvstore-status --verbose
   

3. Verify that the serverVersion and storageEngine fields indicate the latest versions. See the following example:

   serverVersion : 4.2.17
   [...]
   storageEngine : wiredTiger 
   

Set your library path

If your instance fails to automatically migrate to WiredTiger and the latest version of Mongo DB during upgrade, you might need to set or correct your library path. If you receive the following error message, consider setting your library path:

/opt/splunk/bin/mongodump: error while loading shared libraries: libssl.so.1.0.0: cannot open shared object file: No such file or directory

To learn how to set your library path, see About using SSL tools on Windows and Linux in the Securing Splunk Enterprise manual.

Migrate manually to the latest version of Mongo DB

If your instance failed at migrating to WiredTiger, complete the steps for a manual migration in Migrate the KV store after an upgrade to Splunk Enterprise 8.1.* or 8.2.* in a single-instance deployment.

If you have successfully migrated to WiredTiger, but not to the latest version of Mongo DB, manually upgrade to the latest version of Mongo DB with the following CLI command:

splunk migrate migrate-kvstore

Migrate the KV store in a clustered deployment

You must manually migrate your KV store storage engine during your upgrade to Splunk Enterprise 9.0 and higher if you have not done so prior to beginning the upgrade. First prepare your deployment, then migrate to WiredTiger. After you verify that the migration is complete, then upgrade to MongoDB version 4.2.

Prepare your deployment to migrate your storage engine

Avoid any of the following actions right before or during migration:

• If you are running any searches on a KV store node when you begin migrating, that search might fail. Searches that start running after you begin migration are not impacted.
• Do not do any heavy writes to the KV store while the migration is in progress.
• Do not add new search heads while the migration is in progress.

Complete the following steps to prepare your deployment before you migrate your storage engine:

1. Plan sufficient time for your upgrade and migration. The time it takes to migrate the KV store storage engine is proportional to the total data in your KV store.
2. (Optional) Back up your KV store data before you begin the migration process. The KV store non-captain nodes are synced from the captain on a rolling basis, one node at a time, and the migration process does not automatically back up KV store data to a separate location.
3. Upgrade to Splunk Enterprise 9.0 or higher. For more information, see How to upgrade Splunk Enterprise in the Installation Manual.
4. After your upgrade completes, Splunk Enterprise prompts you to upgrade your storage engine immediately to WiredTiger.

Use the curl -k -u admin:changeme -X POST https://localhost:8089/services/shcluster/captain/kvmigrate/stop command to stop the migration process at any time.

Initiate your KV store storage engine migration

After you prepare your deployment, initiate your migration.

1. Check that your instance is ready to migrate by using one of the following commands. You can perform this check with either the REST API or with the Splunk Enterprise command-line interface (CLI).
   REST API:

   curl -k -u admin:changeme https://localhost:8089/services/shcluster/captain/kvmigrate/start -d storageEngine=wiredTiger -d isDryRun=true

   CLI:

   splunk start-shcluster-migration kvstore -storageEngine wiredTiger -isDryRun true

2. Resolve any issues blocking migration. Perform the migration only if all checks pass.
3. Initiate the migration from any node with the following command. To select the options for your command, choose if you want to migrate based on a percentage of nodes or based on specific URIs. If you want to migrate specific peers, specify their names and the management port number. If you specify neither option, then all nodes are migrated on a rolling basis one at a time. Initiate the migration only once from any one node. All nodes are automatically migrated after that.

   Option	REST API sample	CLI sample
   By percentage	curl -k -u admin:changeme https://localhost:8089/services/shcluster/captain/kvmigrate/start -X POST -d storageEngine=wiredTiger -d clusterPerc=50	splunk start-shcluster-migration kvstore -storageEngine wiredTiger -clusterPerc 50
   By URIs	curl -k -u admin:changeme https://localhost:8089/services/shcluster/captain/kvmigrate/start -X POST -d storageEngine=wiredTiger -d peersList="https://server1:8089,https://server2:8089,https://server3:8089"	splunk start-shcluster-migration kvstore -storageEngine wiredTiger -peersList "https://server1:8089,https://server2:8089,https://server3:8089"

Monitor and verify your KV store storage engine migration

After your migration is in progress, you can use any of several methods to monitor your migration and verify that it is complete.

• To check which nodes are currently migrating, use the following commands. You can perform this check with either the REST API or with the Splunk Enterprise command-line interface (CLI).
  REST API:

  curl -k -u admin:changeme https://localhost:8089/services/shcluster/captain/kvmigrate/status

  CLI:

  splunk show shcluster-kvmigration-status

• For more information about the status of the upgrade, use the following command:

  splunk show kvstore-status

• To check the progress of the migration of a cluster member, see the KVStoreReplicaSetStats entry in the $SPLUNK_HOME/var/log/introspection/kvstore.log file on *nix, or the %SPLUNK_HOME\var\log\introspection\kvstore.log file on Windows, on that member. This status updates every 30 seconds.

If you backed up your KV store, verify that the migration is successful and then delete the KV store backup data.

Upgrade KV store server to version 4.2

If you have a single-instance deployment, your server version updates to MongoDB version 4.2 automatically. If you have a clustered deployment, however, choose a maintenance window in which to upgrade to MongoDB version 4.2, and then complete the following steps:

1. Check that your instance is ready to migrate with one of the following commands:
   CLI:

   splunk start-shcluster-upgrade kvstore -version 4.2 -isDryRun true

   REST:

   curl -ku admin:changeme -X POST https://localhost:8089/services/shcluster/captain/kvstore-upgrade/start -d version=4.2 -d isDryRun=true

2. Resolve any issues blocking migration, and then perform the migration only if all checks pass. Initiate the migration only once from any one node. All nodes are automatically migrated after that.
3. Use one of the following commands to initiate this upgrade:
   CLI:

   splunk start-shcluster-upgrade kvstore -version 4.2
   

   REST:

   curl -ku admin:changeme -X POST https://localhost:8089/services/shcluster/captain/kvstore-upgrade/start -d version=4.2
   

4. Verify that you have the latest version of the storage engine after upgrade with one of the following commands:
   CLI:

   splunk show kvstore-status --verbose
   

   REST:

   curl -k -u admin:changeme https://localhost:8089/services/shcluster/captain/kvmigrate/status
   

5. Check the output to see that the serverVersion and storageEngine fields indicate the latest versions:

   serverVersion : 4.2.17
   [...]
   storageEngine : wiredTiger 
   

   In Unix operating systems, the latest server version is 4.2.17. In Windows operating systems, the latest server version is 4.2.19.

Remove unsupported binary files for lower server versions

After you complete your migration to the WiredTiger storage engine and server version 4.2, you can choose to remove the unsupported binary files for MongoDB version 3.6. Removing these files is optional. Complete the following steps to remove the files:

1. Verify that you have the latest version of the storage engine with the following command:

   splunk show kvstore-status --verbose
   

2. Verify that the serverVersion and storageEngine fields indicate the latest versions:

   serverVersion : 4.2.17
   [...]
   storageEngine : wiredTiger 
   

   In Unix operating systems, the latest server version is 4.2.17. In Windows operating systems, the latest server version is 4.2.22.

3. Delete the following files from the $SPLUNK_HOME/bin directory:
   • mongod-3.6
   • mongod-4.0
   • mongodump-3.6
   • mongorestore-3.6

After you remove these files and restart your instance, you can ignore the following message. You don't need to take any action.

03-25-2022 12:34:18.203 -0700 WARN  InstalledFilesHashChecker [3769773 LazyGlobalManifestCheck] - An installed file="/opt/splunk/bin/mongod-3.6" did not pass hash-checking due to reason="file missing"