Splunk® Enterprise

Admin Manual

Download manual as PDF

Download topic as PDF

Resync the KV store

When a KV store member fails to transform its data with all of the write operations, then the KV store member might be stale. To resolve this issue, you must resynchronize the member.

Identify a stale KV store member

You can check the status of the KV store using the command line.

  1. Log into the shell of any KV store member.
  2. Navigate to the bin subdirectory in the Splunk Enterprise installation directory.
  3. Type ./splunk show kvstore-status. The command line returns a summary of the KV store member you are logged into, as well as information about every other member in the KV store cluster.
  4. Look at the replicationStatus field and identify any members that have neither "KV store captain" nor "Non-captain KV store member" as values.

Resync stale KV store members

If more than half of the members are stale, you can either recreate the cluster or resync it from one of the members. See Back up KV store for details about restoring from backup.

To resync the cluster from one of the members, use the following procedure. This procedure triggers the recreation of the KV store cluster, when all of the members of current existing KV store cluster resynchronize all data from the current member (or from the member specified in -source sourceId). The command to resync the KV store cluster can be invoked only from the node that is operating as search head cluster captain.

  1. Determine which node is currently the search head cluster captain. Use the CLI command splunk show shcluster-status.
  2. Log into the shell on the search head cluster captain node.
  3. Run the command splunk resync kvstore [-source sourceId]. The source is an optional parameter, if you want to use a member other than the search head cluster captain as the source.
  4. Enter your admin login credentials.
  5. Wait for a confirmation message on the command line.
  6. Use the splunk show kvstore-status command to verify that the cluster is resynced.


If fewer than half of the members are stale, resync each member individually.

  1. Stop the search head that has the stale KV store member.
  2. Run the command splunk clean kvstore --local.
  3. Restart the search head. This triggers the initial synchronization from other KV store members.
  4. Run the command splunk show kvstore-status to verify synchronization.

Prevent stale members by increasing operations log size

If you find yourself resyncing KV store frequently because KV store members are transitioning to stale mode frequently (daily or maybe even hourly), this means that apps or users are writing a lot of data to the KV store and the operations log is too small. Increasing the size of the operations log (or oplog) might help.

After initial synchronization, noncaptain KV store members no longer access the captain collection. Instead, new entries in the KV store collection are inserted in the operations log. The members replicate the newly inserted data from there. When the operations log reaches its allocation (1 GB by default), it overwrites the beginning of the oplog. Consider a lookup that is close to the size of the allocation. The KV store rolls the data (and overwrites starting from the beginning of the oplog) only after the majority of the members have accessed it, for example, three out of five members in a KV store cluster. But once that happens, it rolls, so a minority member (one of the two remaining members in this example) cannot access the beginning of the oplog. Then that minority member becomes stale and need to be resynced, which means reading from the entire collection (which is likely much larger than the operations log).

To decide whether to increase the operations log size, visit the Monitoring Console KV store: Instance dashboard or use the command line as follows:

  1. Determine which node is currently the search head cluster captain. From any KV store member node, use the CLI command splunk show shcluster-status.
  2. Log into the shell on the search head cluster captain node.
  3. Navigate to $SPLUNK_HOME/bin.
  4. Run splunk show kvstore-status.
  5. Compare the oplog start and end timestamps. The start is the oldest change, and the end is the newest one. If the difference is on the order of a minute, you should probably increase the operations log size.


While keeping your operations log too small has obvious negative effects (like members becoming stale), setting an oplog size much larger than your needs might not be ideal either. The KV store takes the full log size that you allocate right away, regardless of how much data is actually being written to the log. Reading the oplog can take a fair bit of RAM, too, although it is loosely bound. Work with Splunk Support to determine an appropriate operations log size for your KV store use. The operations log is 1 GB by default.

To increase the log size:

  1. Determine which node is currently the search head cluster captain. Use the CLI command splunk show shcluster-status.
  2. Log into the shell on the search head cluster captain node.
  3. Edit server.conf / [kvstore] / oplogSize in the appropriate directory (not default). The default value is 1000 (in units of MB). See How to edit a configuration file.
  4. For each search head:
    1. Log into the search head.
    2. Stop Splunk Enterprise on the search head.
    3. Run the command splunk clean kvstore --local.
    4. Restart the search head. This triggers the initial synchronization from other KV store members.
    5. Run the command splunk show kvstore-status to verify synchronization.
PREVIOUS
About the app key value store
  NEXT
Back up KV Store

This documentation applies to the following versions of Splunk® Enterprise: 6.5.0, 6.5.1, 6.5.1612 (Splunk Cloud only), 6.5.2, 6.5.3, 6.5.4, 6.5.5, 6.6.0, 6.6.1, 6.6.2, 6.6.3, 7.0.0


Was this documentation topic helpful?

Enter your email address, and someone from the documentation team will respond to you:

Please provide your comments here. Ask a question or make a suggestion.

You must be logged into splunk.com in order to post comments. Log in now.

Please try to keep this discussion focused on the content covered in this documentation topic. If you have a more general question about Splunk functionality or are experiencing a difficulty with Splunk, consider posting a question to Splunkbase Answers.

0 out of 1000 Characters