Splunk® Enterprise

Admin Manual

Upgrade the KV store server version

Splunk Enterprise version 9.4 and higher require KV store server version 7.0 or higher. This is a departure from Splunk Enterprise versions 9.0.x through 9.3.x, which require server version 4.2 only.

During your deployment's upgrade to Splunk Enterprise 9.4 or higher, Splunk Enterprise automatically upgrades the KV store server version to 7.0. However, prior to upgrade, you can configure your deployment to prevent the automatic server version upgrade. If you choose this upgrade path, then you must immediately upgrade the server version manually after completing your upgrade to Splunk Enterprise 9.4.x.

Work through this process in this order:

  1. Complete the Prerequisites prior to upgrading to Splunk Enterprise version 9.4.x or higher, regardless of your intended upgrade path.
  2. Choose one of the following options:
  3. If necessary, Troubleshoot a failed upgrade to server version 7.0.
  4. Complete your upgrade to Splunk Enterprise 9.4.x if you have not already done so. For more information about this upgrade, see How to upgrade Splunk Enterprise in the Installation Manual.

Prerequisites

Complete the following steps before upgrading Splunk Enterprise to ensure your upgrade of Splunk Enterprise and the KV store server version go smoothly.

  1. You must upgrade to server version 4.2.x before upgrading to Splunk Enterprise 9.4.x or higher. For instructions and information about updating to KV store server version 4.2.x in Splunk Enterprise versions 9.0.x through 9.3.x, see Migrate the KV store storage engine in the Splunk Enterprise 9.3.0 documentation.
  2. Ensure that more than 50% of your disk space is available.
  3. Confirm that the KV store is healthy by checking its status with the splunk show kvstore-status --verbose command in the CLI. If your KV store is not healthy, locate the [kvstore] stanza of your server.conf file and set the kvstoreUpgradeOnStartupEnabled option to false, then file a case using the Splunk Support Portal for help upgrading your deployment. See Support and Services.
  4. If you are using a clustered deployment, ensure that the cluster is healthy before upgrading your deployment. Use the ./splunk show shcluster-status --verbose command in the CLI to confirm the following items:
    • No nodes are in manual detection mode.
    • No nodes are in maintenance mode.
    • No rolling upgrades or restarts are in progress.
    • The captain is stabilized and not frequently switching.
  5. Check your server.conf file to ensure that you aren't using a custom certificate for the KV store or an IPv6 configuration. Compare with the following examples of custom certificates and IPv6 configurations, located in server.conf: 
    [kvstore]
sslVerifyServerCert = true
sslVerifyServerName = true
caCertFile = $SPLUNK_HOME/<path_to_ca_pem>
serverCert = $SPLUNK_HOME/<path_to_server_pem>
sslPassword = <PASSWORD>

    [general]
listenOnIPv6 = only / yes

    If you are using a custom certificate or IPv6 configuration, your upgrade to server version 7.0 will fail. Upgrades for KV store deployments with custom certificates and IPv6 configurations will be available in future releases of Splunk Enterprise. To work around this issue, you can revert to a default certificate or turn off your IPv6 configuration.

  6. Take a backup of the KV store before initiating your Splunk Enterprise upgrade. For more information about taking a backup of your KV store, see Back up and restore KV store.

    After upgrading Splunk Enterprise but before updating the server version, it is not necessary to take another backup. Splunk Enterprise automatically takes a backup of the KV store at this time. If the KV store server version upgrade fails, Splunk Enterprise automatically restores the KV store from the last backup taken before server version upgrade.

Prepare for a temporary impact on your Splunk Enterprise deployment

The KV store server version upgrade has a temporary impact on your deployment while the upgrade is ongoing. KV store administrator operations are unavailable during this time. Administrator operation include the KV store maintenance mode, restarting the KV store, resyncing the KV store, backing up or restoring the KV store, and any CRUD operations.

Do not perform any heavy writes to the KV store during either the automatic or manual upgrade processes. Writes to the KV store performed during upgrade are not saved in the event of a rollback.

Automatically upgrade to KV store server version 7.0

If you are using KV store server version 4.2.x at the time of your upgrade to Splunk Enterprise 9.4 or higher, then Splunk Enterprise automatically upgrades your deployment to server version 7.0 by default. Complete the following steps to verify your server version is upgraded.

  1. Complete any prompts during your Splunk Enterprise upgrade.
  2. Verify that you have the latest version of the KV store server version after the upgrade with the following command, either in the command-line interface (CLI) or through the REST API:
    CLI: 
    splunk show kvstore-status --verbose

    REST: 

    curl -k -u admin:changeme https://localhost:8089/services/kvstore/status
  3. Check the output to see that it indicates the latest server version:
    CLI: 
    serverVersion : 7.0.14

    REST: 

    version : 7.0.14

Manually upgrade to KV store server version 7.0

If you don't want your KV store server version to automatically upgrade to 7.0 at the same time as your upgrade to Splunk Enterprise 9.4.x, you can turn off the automatic upgrade. If you choose this upgrade path, you must manually upgrade to server version 7.0 immediately after upgrading to Splunk Enterprise 9.4.x.

Complete the following steps to prepare to manually upgrade your KV store server version.

  1. Before initiating the upgrade to Splunk Enterprise 9.4.x, locate the [kvstore] stanza of the server.conf file and set the kvstoreUpgradeOnStartupEnabled option to false.
  2. Determine your deployment type. If your single instance of the KV store is located on a search head, the cluster manager, or any indexer node, you have a single-instance KV store deployment. If you have multiple KV store nodes across a search head cluster, then you have a clustered KV store deployment.
  3. Upgrade to Splunk Enterprise 9.4.x. For more information about completing this upgrade, see How to upgrade Splunk Enterprise in the Installation Manual.
  4. Complete the steps in one of the two following sections, choosing according to your deployment type.

Manually upgrade the KV store server version in a single-instance deployment

Complete the following steps after upgrading to Splunk Enterprise 9.4.x.

  1. Check that your instance is ready to migrate with one of the following commands, either in the CLI or through the REST API:
    CLI: 
    splunk start-standalone-upgrade kvstore -version 7.0 -dryRun true

    REST: 

    curl -ku admin:changeme -X POST https://localhost:8089/services/kvstore/version/upgrade -d version=7.0 -d dryRun=true
  2. Resolve any issues blocking the upgrade, and then perform the upgrade only if all checks pass.
  3. Use one of the following commands to initiate this upgrade:
    CLI: 
    splunk start-standalone-upgrade kvstore -version 7.0

    REST: 

    curl -ku admin:changeme -X POST https://localhost:8089/services/kvstore/version/upgrade -d version=7.0
  4. Track the status of the in-progress upgrade with one of the following commands:
    CLI: 
    splunk show standalone-kvupgrade-status

    REST: 

    curl -ku admin:changeme https://localhost:8089/services/kvstore/version
  5. Verify that you have the latest version of the KV store server version after upgrade with one of the following commands:
    CLI: 
    splunk show kvstore-status --verbose

    REST: 

    curl -k -u admin:changeme https://localhost:8089/services/kvstore/status
  6. Check that the output indicates the latest server version:
    CLI: 
    serverVersion : 7.0.14

    REST: 

    version : 7.0.14

Manually upgrade the KV store server version in a clustered deployment

Complete the following steps after upgrading to Splunk Enterprise 9.4.x.

  1. Check that your instance is ready to upgrade with one of the following commands, either in the CLI or through the REST API:
    CLI: 
    splunk start-shcluster-upgrade kvstore -version 7.0 -isDryRun true

    REST: 

    curl -ku admin:changeme -X POST https://localhost:8089/services/shcluster/captain/kvstore-upgrade/start -d version=7.0 -d isDryRun=true
  2. Resolve any issues blocking the upgrade, and then perform the upgrade only if all checks pass. Initiate the upgrade only once from any one node. All nodes are automatically upgraded after that.
  3. Use one of the following commands to initiate this upgrade:
    CLI: 
    splunk start-shcluster-upgrade kvstore -version 7.0

    REST: 

    curl -ku admin:changeme -X POST https://localhost:8089/services/shcluster/captain/kvstore-upgrade/start -d version=7.0
  4. Track the status of the in-progress upgrade with one of the following commands:
    CLI: 
    splunk show kvstore-upgrade-status

    REST: 

    curl -ku admin:changeme https://localhost:8089/services/shcluster/captain/kvstore-upgrade/status

    Stop an in-progress upgrade at any time with the following REST API command:

    curl -X POST -k -u admin:changeme https://localhost:8089/services/shcluster/captain/kvstore-upgrade/stop

  5. Verify that you have the latest version of the KV store server version after the upgrade with one of the following commands:
    CLI: 
    splunk show kvstore-status --verbose

    REST: 

    curl -k -u admin:changeme https://localhost:8089/services/shcluster/captain/kvstore-upgrade/status
  6. Check that the output indicates the latest server version:
    CLI: 
    serverVersion : 7.0.14

    REST: 

    version : 7.0.14

Troubleshoot a failed upgrade to server version 7.0

If your upgrade to KV store server version 7.0 fails, complete the following steps to begin troubleshooting the problem.

  1. Ensure you completed the steps outlined in the Prerequisites section.
  2. File a case using the Splunk Support Portal for help upgrading your deployment. See Support and Services.
  3. To avoid triggering an attempt at an automatic upgrade every time your restart your Splunk deployment, locate the [kvstore] stanza of your server.conf file and set the kvstoreUpgradeOnStartupEnabled option to false. While this option is set to false, Splunk Enterprise still reminds you of this pending upgrade every time you restart.
Last modified on 17 December, 2024
Back up and restore KV store   KV store troubleshooting tools

This documentation applies to the following versions of Splunk® Enterprise: 9.4.0

Acrobat logo Download manual
Acrobat logo Download this page

Was this topic useful?







You must be logged into splunk.com in order to post comments. Log in now.

Please try to keep this discussion focused on the content covered in this documentation topic. If you have a more general question about Splunk functionality or are experiencing a difficulty with Splunk, consider posting a question to Splunkbase Answers.

0 out of 1000 Characters