Splunk® Enterprise

Monitoring Splunk Enterprise

Acrobat logo Download manual as PDF

Splunk Enterprise version 6.x is no longer supported as of October 23, 2019. See the Splunk Software Support Policy for details. For information about upgrading to a supported version, see How to upgrade Splunk Enterprise.
This documentation does not apply to the most recent version of Splunk. Click here for the latest version.
Acrobat logo Download topic as PDF

KV store: Deployment

This topic is a reference for the KV store: Deployment dashboard in the distributed management console. See “About the distributed management console.”

What does this view show?

The KV store: Deployment view in the distributed management console (DMC) provides information aggregated across all KV stores in your Splunk Enterprise deployment. For an instance to be included in this view, it must be set with the server role of KV store. Do this in the DMC Setup page.

This view and the KV store: Instance view track much of the same information. The difference is that this deployment view collects statistics from KV stores and displays the instances grouped by values of those different metrics.

For definitions and context on the individual dashboards and metrics, see "KV store: instance" in this chapter.

Performance Metrics

Deployment Snapshots

Deployment Snapshot Statistics access the /services/server/introspection/kvstore/serverstatus REST endpoint.

Where does this view get its data from?

KV store collects data in the _introspection index.

These statistics are broken into the following components:

  • KVStoreServerStats. Information about how the KV store process is performing as a whole. Polled every 27 seconds.
  • KVStoreCollectionStats. Information about collections within the KV store. Polled every 10 minutes.
  • KVStoreReplicaSetStats. Information about replication data across KV store Instances. Polled every 60 seconds.
  • KVProfilingStats. Information about slow operations. Polled every 5 seconds. Only available when profiling is enabled. Note: Enable profile only on development systems or for troubleshooting issues with KV store performance beyond what is available in the default panels. Profiling can negatively affect system performance and so should not be enabled in production environments.

In addition, KV store produces entries in a number of internal logs collected by Splunk Enterprise.

Interpret this view

Panel Critical Warning Normal Interpretation
Page faults per operation 1.3+

Reads require heavy disk I/O, which could indicate a need for more RAM.


Reads regularly require disk I/O.


Reads rarely require disk I/O.

Measures how often read requests are not satisfied by what Splunk Enterprise has in memory, requiring Splunk Enterprise to contact the disk. Windows counts "soft page faults," so Windows machines exhibit more page faults. Consult Lock percentage and Queues instead for Windows.
Lock percentage 50%+ 30%–50% 0–30% High lock percentage can starve replication and/or cause application calls to be slow, time out, or fail. High lock percentage typically means that heavy write activity is occurring on the node.
Network traffic N/A N/A N/A Network traffic should be commensurate with system use and application expectations. No default thresholds apply.
Replication latency >30 seconds 10–30 seconds 0–10 seconds Replication needs are system dependent. Generally, replica set members should not fall significantly behind the KV captain. Replication latency over 30 seconds can indicate a mounting replication problem.
Primary operations log window N/A N/A N/A Provided for reference. This is the amount of data, in terms of time, a system saved in the operations log for restoration.
Flushing rate 50%–100% 10%–50% 0–10% A high flush rate indicates heavy write operations or sluggish system performance.

Troubleshoot this view

The historical panels get data from the _introspection and _internal indexes. Gaps in time in these panels indicate a point at which KV store or Splunk Enterprise was unreachable. If a panel is completely blank or missing data from specific Splunk Enterprise instances, check:

Last modified on 27 September, 2016
KV store: Instance
Search head clustering dashboards

This documentation applies to the following versions of Splunk® Enterprise: 6.3.0, 6.3.1, 6.3.2, 6.3.3, 6.3.4, 6.3.5, 6.3.6, 6.3.7, 6.3.8, 6.3.9, 6.3.10, 6.3.11, 6.3.12, 6.3.13, 6.3.14, 6.4.0, 6.4.1, 6.4.2, 6.4.3, 6.4.4, 6.4.5, 6.4.6, 6.4.7, 6.4.8, 6.4.9, 6.4.10, 6.4.11

Was this documentation topic helpful?

You must be logged into splunk.com in order to post comments. Log in now.

Please try to keep this discussion focused on the content covered in this documentation topic. If you have a more general question about Splunk functionality or are experiencing a difficulty with Splunk, consider posting a question to Splunkbase Answers.

0 out of 1000 Characters