KV store troubleshooting tools
This topic discusses tools for viewing KV store status and its log files. It also discusses some monitoring tools that you can use in Splunk Enterprise.
Check KV store status
You can check the status of the KV store in the following ways:
- Use the command line.
- Make a REST API GET request.
- Run the KV store health check in the monitoring console. See Access and customize health check in Monitoring Splunk Enterprise.
KV store status CLI command
On the command line from any KV store member, in $SPLUNK_HOME/bin type the following command:
./splunk show kvstore-status
See About the CLI for information about using the CLI in Splunk software.
KV store status REST endpoint
Use cURL to make a GET request with the REST API:
curl -k -u user:pass https://<host>:<mPort>/services/kvstore/status
See Basic Concepts in the REST API User Manual for more information about the REST API.
KV store status definitions
The following is a list of possible values for status and replicationStatus and their definitions. For more information about abnormal statuses for your KV store members, check mongod.log and splunkd.log for errors and warnings.
| KV store status | Definition |
|---|---|
| starting |
|
| disabled | KV store is disabled in server.conf on this instance. If this member is a search head cluster member, its status remains disabled only if all other members of the search head cluster have KV store disabled. |
| ready | KV store is ready for use. |
| failed | Failed to bootstrap and join the search head cluster. |
| shuttingdown | Splunk software has notified KV store about the shutting down procedure. |
| KV store replication status | Definition |
|---|---|
| Startup | Member is starting. |
| KV store captain | Member is the elected KV store captain. |
| Non-captain KV store member | Healthy noncaptain member of KV store cluster. |
| Initial sync | This member is resynchronizing data from one of the other KV store cluster members. If this happens often, or if this member remains in this state, check mongod.log and splunkd.log on this member, and verify connection to this member and connection speed. |
| Down | Member is stopped. |
| Removed | Member is removed from the KV store cluster, or is in the process of being removed. |
| Rollback / Recovering / Unknown status | Member might have a problem. Check mongod.log and splunkd.log on this member. |
Sample command-line response:
This member: date : Tue Jul 21 16:42:24 2016 dateSec : 1466541744.143000 disabled : 0 guid : 6244DF36-D883-4D59-AHD3-5276FCB4BL91 oplogEndTimestamp : Tue Jul 21 16:41:12 2016 oplogEndTimestampSec : 1466541672.000000 oplogStartTimestamp : Tue Jul 21 16:34:55 2016 oplogStartTimestampSec : 1466541295.000000 port : 8191 replicaSet : splunkrs replicationStatus : KV store captain standalone : 0 status : ready Enabled KV store members: 10.140.137.128:8191 guid : 6244DF36-D883-4D59-AHD3-5276FCB4BL91 hostAndPort : 10.140.137.128:8191 10.140.137.119:8191 guid : 8756FA39-F207-4870-BC5D-C57BABE0ED18 hostAndPort : 10.140.137.119:8191 10.140.136.112:8191 guid : D6190F30-C59A-423Q-AB48-80B0012317V5 hostAndPort : 10.140.136.112:8191 KV store members: 10.140.137.128:8191 configVersion : 1 electionDate : Tue Jul 21 16:42:02 2016 electionDateSec : 1466541722.000000 hostAndPort : 10.140.134.161:8191 optimeDate : Tue Jul 21 16:41:12 2016 optimeDateSec : 1466541672.000000 replicationStatus : KV store captain uptime : 108 10.140.137.119:8191 configVersion : 1 hostAndPort : 10.140.134.159:8191 lastHeartbeat : Tue Jul 21 16:42:22 2016 lastHeartbeatRecv : Tue Jul 21 16:42:22 2016 lastHeartbeatRecvSec : 1466541742.490000 lastHeartbeatSec : 1466541742.937000 optimeDate : Tue Jul 21 16:41:12 2016 optimeDateSec : 1466541672.000000 pingMs : 0 replicationStatus : Non-captain KV store member uptime : 107 10.140.136.112:8191 configVersion : -1 hostAndPort : 10.140.133.82:8191 lastHeartbeat : Tue Jul 21 16:42:22 2016 lastHeartbeatRecv : Tue Jul 21 16:42:00 2016 lastHeartbeatRecvSec : 1466541720.503000 lastHeartbeatSec : 1466541742.959000 optimeDate : ZERO_TIME optimeDateSec : 0.000000 pingMs : 0 replicationStatus : Down uptime : 0
KV store messages
The KV store logs error and warning messages in internal logs, including splunkd.log and mongod.log. These error messages post to the bulletin board in Splunk Web. See What Splunk software logs about itself for an overview of internal log files.
Recent KV store error messages also appear in the REST /services/messages endpoint. You can use cURL to make a GET request for the endpoint, as follows:
curl -k -u user:pass https://<host>:<mPort>/services/messages
For more information about introspection endpoints, see System endpoint descriptions in the REST API Reference Manual.
KV store migration message
If you experience migration issues with using the KV store, then the following lines appear in the mongod.log file:
2018-07-17T15:44:12.122-0700 F STORAGE [initandlisten] BadValue: Invalid value for version, found 3.2, expected '3.6' or '3.4'. Contents of featureCompatibilityVersion document in admin.system.version: { _id: "featureCompatibilityVersion", version: "3.2" }. See http://dochub.mongodb.org/core/3.6-feature-compatibility.
2018-07-17T15:44:12.122-0700 F CONTROL [initandlisten] ** IMPORTANT: UPGRADE PROBLEM: Unable to parse the featureCompatibilityVersion document. The data files need to be fully upgraded to version 3.4 before attempting an upgrade to 3.6. If you are upgrading to 3.6, see http://dochub.mongodb.org/core/3.6-upgrade-fcv.
If you see these lines, enter the splunk migrate migrate-kvstore command to complete the migration.
Downgrading to Splunk Enterprise version 7.1 from version 7.2 causes an error in the mongod.log file.
If you downgrade to Splunk Enterprise version 7.1 from version 7.2, you might receive the following error in mongod.log:
2018-07-17T15:49:23.035-0700 I - [initandlisten] Fatal assertion 18523 InvalidOptions: The field 'uuid' is not a valid collection option. Options: { capped: true, size: 10485760, uuid: BinData(4, 3EC1315074984FEC94A1AE35848760B6) } at src/mongo/db/storage/mmap_v1/mmap_v1_database_catalog_entry.cpp 901
2018-07-17T15:49:23.035-0700 I - [initandlisten]
***aborting after fassert() failure
2018-07-17T15:49:23.043-0700 F - [initandlisten] Got signal: 6 (Abort trap: 6).
Before downgrading from Splunk Enterprise version 7.2 to 7.1, resync the KV store with the following command:
curl -u username:password -XPOST https://localhost:8089/services/kvstore/resync/resync?featureCompatibilityVersion=3.4
If you use this command and and then restart Splunk before downgrading, run this command again before downgrading.
Updating the IP address of a KV store server can require a resync
If you update the IP address of a KV store server, you might receive the following error in mongod.log:
Did not find local replica set configuration document at startup; NoMatchingDocument Did not find replica set configuration document in local.system.replset
To reconfigure the cluster to pick up the new IP address, resync to force the cluster configuration to refresh:
splunk resync shcluster-replicated-config
A manual resync with this command overwrites any local changes on that KV store server. For more information about manually resyncing a cluster member, see Why a recovering member might need to resync manually in the Distributed Search manual.
For more information about resyncing the KV store, see Resync the KV store.
Monitor KV store performance
You can monitor your KV store performance through two views in the monitoring console. The KV store: Deployment dashboard provides information aggregated across all KV stores in your Splunk Enterprise deployment. The KV store: Instance dashboard shows performance information about a single Splunk Enterprise instance running the KV store. See KV store dashboards in Monitoring Splunk Enterprise.
| Back up and restore KV store | Apps and add-ons |
This documentation applies to the following versions of Splunk® Enterprise: 7.2.0, 7.2.1
Download manual
Feedback submitted, thanks!