Splunk® Enterprise

Admin Manual

Acrobat logo Download manual as PDF


Splunk Enterprise version 8.0 is no longer supported as of October 22, 2021. See the Splunk Software Support Policy for details. For information about upgrading to a supported version, see How to upgrade Splunk Enterprise.
This documentation does not apply to the most recent version of Splunk® Enterprise. For documentation on the most recent version, go to the latest release.
Acrobat logo Download topic as PDF

KV store troubleshooting tools

This topic discusses tools for viewing KV store status and its log files. It also discusses some monitoring tools that you can use in Splunk Enterprise.

Check KV store status

You can check the status of the KV store in the following ways:

  • Use the command line.
  • Make a REST API GET request.
  • Run the KV store health check in the monitoring console. See Access and customize health check in Monitoring Splunk Enterprise.

KV store status CLI command

On the command line from any KV store member, in $SPLUNK_HOME/bin type the following command:

./splunk show kvstore-status

See About the CLI for information about using the CLI in Splunk software.

KV store status REST endpoint

Use cURL to make a GET request with the REST API:

curl -k -u user:pass https://<host>:<mPort>/services/kvstore/status

See Basic Concepts in the REST API User Manual for more information about the REST API.

KV store status definitions

The following is a list of possible values for status and replicationStatus and their definitions. For more information about abnormal statuses for your KV store members, check mongod.log and splunkd.log for errors and warnings.

KV store status Definition
starting
  • In the case of a standalone search head, this status switches to ready after synchronization of a list of defined collections, accelerated fields, and so on.
  • In the case of a search head cluster, this status switches to ready when the search head cluster is bootstrapped (after the search head cluster captain is elected) and the search head cluster captain propagates status to all search head cluster members.
disabled KV store is disabled in server.conf on this instance. If this member is a search head cluster member, its status remains disabled only if all other members of the search head cluster have KV store disabled.
ready KV store is ready for use.
failed Failed to bootstrap and join the search head cluster.
shuttingdown Splunk software has notified KV store about the shutting down procedure.
KV store replication status Definition
Startup Member is starting.
KV store captain Member is the elected KV store captain.
Non-captain KV store member Healthy noncaptain member of KV store cluster.
Initial sync This member is resynchronizing data from one of the other KV store cluster members. If this happens often, or if this member remains in this state, check mongod.log and splunkd.log on this member, and verify connection to this member and connection speed.
Down Member is stopped.
Removed Member is removed from the KV store cluster, or is in the process of being removed.
Rollback / Recovering / Unknown status Member might have a problem. Check mongod.log and splunkd.log on this member.

Sample command-line response:

This member:
		                     date : Tue Jul 21 16:42:24 2016
		                  dateSec : 1466541744.143000
		                 disabled : 0
		                     guid : 6244DF36-D883-4D59-AHD3-5276FCB4BL91
		        oplogEndTimestamp : Tue Jul 21 16:41:12 2016
		     oplogEndTimestampSec : 1466541672.000000
		      oplogStartTimestamp : Tue Jul 21 16:34:55 2016
		   oplogStartTimestampSec : 1466541295.000000
		                     port : 8191
		               replicaSet : splunkrs
		        replicationStatus : KV store captain
		               standalone : 0
		                   status : ready

 Enabled KV store members:
	10.140.137.128:8191
		                     guid : 6244DF36-D883-4D59-AHD3-5276FCB4BL91
		              hostAndPort : 10.140.137.128:8191
	10.140.137.119:8191
		                     guid : 8756FA39-F207-4870-BC5D-C57BABE0ED18
		              hostAndPort : 10.140.137.119:8191
	10.140.136.112:8191
		                     guid : D6190F30-C59A-423Q-AB48-80B0012317V5
		              hostAndPort : 10.140.136.112:8191

 KV store members:
	10.140.137.128:8191
		            configVersion : 1
		             electionDate : Tue Jul 21 16:42:02 2016
		          electionDateSec : 1466541722.000000
		              hostAndPort : 10.140.134.161:8191
		               optimeDate : Tue Jul 21 16:41:12 2016
		            optimeDateSec : 1466541672.000000
		        replicationStatus : KV store captain
		                   uptime : 108
	10.140.137.119:8191
		            configVersion : 1
		              hostAndPort : 10.140.134.159:8191
		            lastHeartbeat : Tue Jul 21 16:42:22 2016
		        lastHeartbeatRecv : Tue Jul 21 16:42:22 2016
		     lastHeartbeatRecvSec : 1466541742.490000
		         lastHeartbeatSec : 1466541742.937000
		               optimeDate : Tue Jul 21 16:41:12 2016
		            optimeDateSec : 1466541672.000000
		                   pingMs : 0
		        replicationStatus : Non-captain KV store member
		                   uptime : 107
	10.140.136.112:8191
		            configVersion : -1
		              hostAndPort : 10.140.133.82:8191
		            lastHeartbeat : Tue Jul 21 16:42:22 2016
		        lastHeartbeatRecv : Tue Jul 21 16:42:00 2016
		     lastHeartbeatRecvSec : 1466541720.503000
		         lastHeartbeatSec : 1466541742.959000
		               optimeDate : ZERO_TIME
		            optimeDateSec : 0.000000
		                   pingMs : 0
		        replicationStatus : Down
		                   uptime : 0

KV store messages

The KV store logs error and warning messages in internal logs, including splunkd.log and mongod.log. These error messages post to the bulletin board in Splunk Web. See What Splunk software logs about itself for an overview of internal log files.

Recent KV store error messages also appear in the REST /services/messages endpoint. You can use cURL to make a GET request for the endpoint, as follows:

curl -k -u user:pass https://<host>:<mPort>/services/messages

For more information about introspection endpoints, see System endpoint descriptions in the REST API Reference Manual.


KV store migration message

If you experience migration issues with using the KV store, then the following lines appear in the mongod.log file:

2018-07-17T15:44:12.122-0700 F STORAGE [initandlisten] BadValue: Invalid value for version, found 3.2, expected '3.6' or '3.4'. Contents of featureCompatibilityVersion document in admin.system.version: { _id: "featureCompatibilityVersion", version: "3.2" }. See http://dochub.mongodb.org/core/3.6-feature-compatibility.

2018-07-17T15:44:12.122-0700 F CONTROL [initandlisten] ** IMPORTANT: UPGRADE PROBLEM: Unable to parse the featureCompatibilityVersion document. The data files need to be fully upgraded to version 3.4 before attempting an upgrade to 3.6. If you are upgrading to 3.6, see http://dochub.mongodb.org/core/3.6-upgrade-fcv.

If you see these lines, complete the following steps to migrate the KV store manually:

  1. Enter the splunk migrate mongod-fix-voting-priority command.

    Do not proceed to the next step unless this command completes successfully.

  2. Enter the splunk migrate migrate-kvstore command.

Updating the IP address of a KV store server can require a resync

If you update the IP address of a KV store server, you might receive the following error in mongod.log:

Did not find local replica set configuration document at startup; NoMatchingDocument 
Did not find replica set configuration document in local.system.replset

To reconfigure the cluster to pick up the new IP address, resync to force the cluster configuration to refresh:

splunk resync shcluster-replicated-config

A manual resync with this command overwrites any local changes on that KV store server. For more information about manually resyncing a cluster member, see Why a recovering member might need to resync manually in the Distributed Search manual.

For more information about resyncing the KV store, see Resync the KV store.

Monitor KV store performance

You can monitor your KV store performance through two views in the monitoring console. The KV store: Deployment dashboard provides information aggregated across all KV stores in your Splunk Enterprise deployment. The KV store: Instance dashboard shows performance information about a single Splunk Enterprise instance running the KV store. See KV store dashboards in Monitoring Splunk Enterprise.

Last modified on 08 June, 2022
PREVIOUS
Back up and restore KV store
  NEXT
Apps and add-ons

This documentation applies to the following versions of Splunk® Enterprise: 7.2.2, 7.2.3, 7.2.4, 7.2.5, 7.2.6, 7.2.7, 7.2.8, 7.2.9, 7.2.10, 7.3.0, 7.3.1, 7.3.2, 7.3.3, 7.3.4, 7.3.5, 7.3.6, 7.3.7, 7.3.8, 7.3.9, 8.0.0, 8.0.1, 8.0.2, 8.0.3, 8.0.4, 8.0.5, 8.0.6, 8.0.7, 8.0.8, 8.0.9, 8.0.10, 8.1.0, 8.1.1, 8.1.2, 8.1.3, 8.1.4, 8.1.5, 8.1.6, 8.1.7, 8.1.8, 8.1.9, 8.1.10, 8.1.11, 8.1.12, 8.1.13, 8.1.14, 8.2.0, 8.2.1, 8.2.2, 8.2.3, 8.2.4, 8.2.5, 8.2.6, 8.2.7, 8.2.8, 8.2.9, 8.2.10, 8.2.11, 8.2.12


Was this documentation topic helpful?


You must be logged into splunk.com in order to post comments. Log in now.

Please try to keep this discussion focused on the content covered in this documentation topic. If you have a more general question about Splunk functionality or are experiencing a difficulty with Splunk, consider posting a question to Splunkbase Answers.

0 out of 1000 Characters