Manage the KV store workload with read-only mode

Read-only mode can temporarily forcibly stop all writes to the KV store. The most likely case in which you might want to block writes to the KV store during the KV store server version upgrade, because heavy writes to the KV store during upgrade can cause it to fail. For more information about the server version upgrade, see Upgrade the KV store server version.

Detect a high workload

In a clustered deployment of Splunk Enteprise, you can determine if you need to block writes to the KV store by detecting the KV store's workload. Complete the following steps to determine if you have a high workload.

1. Prepare the following SPL command by replacing the placeholder text <search-head-member1>, ..., <search-head-member-n> with your list of search head cluster members.

   index="_introspection" component=KVStoreReplicaSetStats "data.replSetStats.myState"=1 data.replSetStats.set=splunkrs host IN (<search-head-member1>, ..., <search-head-member-n>) | spath data.replSetStats.members{}.name output=searchhead | spath data.replSetStats.members{}.stateStr output=state | spath data.replSetStats.members{}.optimeDate output=oplogEpoch | eval prop_key_val=mvzip(mvzip(state, searchhead , "---"), oplogEpoch, "---") | fields _time, prop_key_val | eval primary=mvfilter(match(prop_key_val,"^PRIMARY")) | eval secondary=mvfilter(match(prop_key_val,"^SECONDARY")) | eval primaryOplogEpoch=mvindex(split(primary,"---"),2) | fields _time primaryOplogEpoch, secondary | fields - _raw | mvexpand secondary | rex field=secondary "SECONDARY---(?<searchhead>.*)---(?<oplogEpoch>\d+)" | eval lagSeconds=(primaryOplogEpoch - oplogEpoch)/1000 | timechart span=1m max(lagSeconds) as MaxLagSeconds by searchhead limit=0

2. Run your modified command in the Splunk Enterprise Search app from a deployer that has access to introspection logs on all search head cluster members.
3. Explore the results of your command. If the results show that the lag between the primary and secondary KV store replies is consistently above 10 seconds, then your workload is high.

If these commands confirm that you have a high workload, then use read-only mode to block heavy writes to the KV store while you upgrade the KV store server version. This prevents the upgrade from failing due to heavy writes.

Upgrade the KV store server version in a clustered deployment with read-only mode

Writes to the KV store performed during an upgrade of Splunk Enterprise or the KV store server version are not necessarily saved in the event of a rollback. To prevent failure of the KV store server version upgrade, you can choose to completely block writes to the KV store during the upgrade by using read-only mode.

Complete the following steps to upgrade a clustered deployment of Splunk Enterprise and the KV store using read-only mode.

1. Turn off the automatic upgrade of the KV store. Before initiating the upgrade to Splunk Enterprise 9.4.x, locate the [kvstore] stanza of the server.conf file and set the kvstoreUpgradeOnStartupEnabled option to false.
2. Upgrade to Splunk Enterprise 9.4.x. For more information about completing this upgrade, see How to upgrade Splunk Enterprise in the Installation Manual.
3. Switch to a static KV store captain for your search head cluster. For more information about switching to a static captain, see Switch to a static captain in the Distributed Search manual.
4. Turn on read-only mode on the captain by entering the following line in the command-line interface (CLI).

   splunk enable kvstore-maintenance-mode -readOnly true

5. Upgrade the KV store server version manually, monitoring its progress until it is complete. For more information about manual upgrade, see Manually upgrade the KV store server version in a clustered deployment.
6. Turn off read-only mode with the following command in the CLI.

   splunk disable kvstore-maintenance-mode -readOnly true

7. Revert to a dynamic captain for the search head cluster. For more information about switching back to a dynamic captain, see Revert to the dynamic captain in the Distributed Search manual.

After completing these steps, your Splunk Enterprise and KV store server version upgrades should be complete. To troubleshoot this process or learn more about it, see Upgrade the KV store server version.

Enter and exit read-only mode

You can turn read-only mode on or off at any time by entering the following commands into the CLI. If you have a clustered deployment, enter these commands from the KV store captain.

To turn on read-only mode, use the following command.

splunk enable kvstore-maintenance-mode -readOnly true

To turn on read-only mode, use the following command.

splunk disable kvstore-maintenance-mode -readOnly true