Splunk® Enterprise

Monitoring Splunk Enterprise

Acrobat logo Download manual as PDF

Splunk Enterprise version 6.x is no longer supported as of October 23, 2019. See the Splunk Software Support Policy for details. For information about upgrading to a supported version, see How to upgrade Splunk Enterprise.
This documentation does not apply to the most recent version of Splunk. Click here for the latest version.
Acrobat logo Download topic as PDF

KV store: Instance

This topic is a reference for the KV store: Instance dashboard in the distributed management console. See “About the distributed management console.”

What does this view show?

The instance level KV store view in the distributed management console (DMC) shows performance information about a single Splunk Enterprise instance running the app key-value store. If you have configured the DMC with your distributed deployment, you can select which instance in your deployment to view.

Performance metrics

Collection metrics come from the KVStoreCollectionStats component in the _introspection index, which is a historical record of the data at the /services/server/introspection/kvstore/collectionstats REST endpoint. The metrics are:

  • Application. The application the collection belongs to.
  • Collection. The name of the collection in KV store.
  • Number of objects. The count of data objects stored in collection.
  • Accelerations. The count of accelerations set up on the collection. Note: These are traditional database-style indexes used for performance and search acceleration.
  • Accelerations size. The size in MBs of the indexes set up on the collection.
  • Collection size. The size in MBs of all data stored in the collection.

Snapshots are collected through REST endpoints, which deliver the most recent information from the pertinent introspection components. The KV store instance snapshots use the endpoint /services/server/introspection/kvstore/serverstatus.

  • Lock percentage. The percentage of KV store uptime that the system has held either global read or write locks. A high lock percentage has impacts across the board. It can starve replication or even make application calls slow, time out, or fail.
  • Page fault percentage. The percentage of KV store operations that resulted in a page fault. A percentage close to 1 indicates poor system performance and is a leading indicator of continued sluggishness as KV store is forced to fallback on disk I/O rather than access data store efficiently in memory.
  • Memory usage. The amount of resident, mapped, and virtual memory in use by KV store. Virtual memory usage is typically twice that of mapped memory for KV store. Virtual memory usage in excess of 3X mapped might indicate a memory leak.
  • Network traffic. Total MBs in and out of KV store network traffic.
  • Flush percentage. Percentage of a minute it takes KV store to flush all writes to disk. Closer to 1 indicates difficulty writing to disk or consistent large write operations. Some OSes can flush data faster than 60 seconds. In that case, this number can be small even if there is a writing bottleneck.
  • Operations. Count of operations issued to KV store. Includes commands, updates, queries, deletes, getmores, and inserts. The introspection process issues a command to deliver KV store stats so the commands counter is typically higher than most other operations.
  • Current connections. Count of connections open on KV store.
  • Total queues. Total operations queued waiting for the lock.
  • Total asserts. Total number of asserts raised by KV store. A non-negative number can indicate a need to check KV store logs.


Many of the statistics in this section are present in the Snapshots section. The Historical view presents trend information for the metrics across a set span of time. These stats are collected in KVStoreServerStats. By default the Historical panels show information for the past 4 hours. Any gaps in the graphs in this section typically indicate a point at which KV store or Splunk Enterprise was unreachable.

  • Memory usage - see above.
  • Replication lag. The amount of time between the last operation recorded in the Primary OpLog and the last operation applied to a secondary node. Replication lag in excess of the primary opLog window could result in data not being properly replicated across all nodes of the replication set. In standalone instances without replication this panel does not return any results. Note: Replication lag is collected in the KVStoreReplicaSetStats component in the _introspection index.
  • Operation count (average by minute) - see above. This panel shows individual operation types (for example, commands, updates, and deletes) or for all operations.
  • Asserts - see above. This panel allows for filtering based on type of assert - message, regular, rollovers, user, warning.
  • Lock percentage. Percentage of KV store uptime that the system has held global, read, or write locks. Filter this panel by type of lock held:
    • Read. Lock held for read operations.
    • Write. Lock held for write operations. KV store locking is "writer greedy," so write locks can make up the majority of the total locks on a collection.
    • Global. Lock held by the global system. KV store implements collection-level locks, reducing the need for aggressive use of the global lock.
  • Page faults as a percentage of total operations - see above.
  • Network traffic - see above. Added to this panel are requests made to the KV store.
  • Queues over time. The number of queues, broken down by:
    • Read. Count of read operations waiting for a read lock to open.
    • Write. Count of write operations waiting for a write lock to open.
    • Total.
  • Connections over time.
  • Percent of each minute spent flushing to disk - see above.
  • Slowest operations. The ten slowest operations logged by KV store in the selected time frame. If profiling is off for all collections, this could have no results even if you have very slow operations running. Enable profiling on a per collection basis in collections.conf.

Where does this view get its data from?

KV store collects data in the _introspection index.

These statistics are broken into the following components:

  • KVStoreServerStats. Information about how the KV store process is performing as a whole. Polled every 27 seconds.
  • KVStoreCollectionStats. Information about collections within the KV store. Polled every 10 minutes.
  • KVStoreReplicaSetStats. Information about replication data across KV store Instances. Polled every 60 seconds.
  • KVProfilingStats. Information about slow operations. Polled every 5 seconds. Only available when profiling is enabled. Note: Enable profiling only on development systems or for troubleshooting issues with KV store performance beyond what is available in the default panels. Profiling can negatively affect system performance and so should not be enabled in production environments.

In addition, KV store produces entries in a number of internal logs collected by Splunk Enterprise.

Interpret results in this view

For information on performance indicators and red flags, see "KV store: Deployment" in this manual.

Troubleshoot this view

The historical panels get data from introspection logs. Any gaps in time in the graphs in this section typically indicate a point at which KV store or Splunk Enterprise was unreachable. If a panel is completely blank or missing data from specific Splunk Enterprise instances, check:

Last modified on 02 August, 2016
Resource usage: Deployment
KV store: Deployment

This documentation applies to the following versions of Splunk® Enterprise: 6.3.0, 6.3.1, 6.3.2, 6.3.3, 6.3.4, 6.3.5, 6.3.6, 6.3.7, 6.3.8, 6.3.9, 6.3.10, 6.3.11, 6.3.12, 6.3.13, 6.3.14, 6.4.0, 6.4.1, 6.4.2, 6.4.3, 6.4.4, 6.4.5, 6.4.6, 6.4.7, 6.4.8, 6.4.9, 6.4.10, 6.4.11

Was this documentation topic helpful?

You must be logged into splunk.com in order to post comments. Log in now.

Please try to keep this discussion focused on the content covered in this documentation topic. If you have a more general question about Splunk functionality or are experiencing a difficulty with Splunk, consider posting a question to Splunkbase Answers.

0 out of 1000 Characters