KV store: Deployment

This topic is a reference for the KV store: Deployment dashboard in the distributed management console. See “About the distributed management console.”

What does this view show?

The KV store: Deployment view in the distributed management console (DMC) provides information aggregated across all KV stores in your Splunk Enterprise deployment. For an instance to be included in this view, it must be set with the server role of KV store. Do this in the DMC Setup page.

This view and the KV store: Instance view track much of the same information. The difference is that this deployment view collects statistics from KV stores and displays the instances grouped by values of those different metrics.

For definitions and context on the individual dashboards and metrics, see "KV store: instance" in this chapter.

Performance Metrics

Deployment Snapshots

Deployment Snapshot Statistics access the /services/server/introspection/kvstore/serverstatus REST endpoint.

Where does this view get its data from?

KV store collects data in the _introspection index.

These statistics are broken into the following components:

• KVStoreServerStats. Information about how the KV store process is performing as a whole. Polled every 27 seconds.
• KVStoreCollectionStats. Information about collections within the KV store. Polled every 10 minutes.
• KVStoreReplicaSetStats. Information about replication data across KV store Instances. Polled every 60 seconds.
• KVProfilingStats. Information about slow operations. Polled every 5 seconds. Only available when profiling is enabled. Note: Enable profile only on development systems or for troubleshooting issues with KV store performance beyond what is available in the default panels. Profiling can negatively affect system performance and so should not be enabled in production environments.

In addition, KV store produces entries in a number of internal logs collected by Splunk Enterprise.

Interpret this view

Panel	Critical	Warning	Normal	Interpretation
Page faults per operation	1.3+ Reads require heavy disk I/O, which could indicate a need for more RAM.	0.7–1.3 Reads regularly require disk I/O.	0–0.7 Reads rarely require disk I/O.	Measures how often read requests are not satisfied by what Splunk Enterprise has in memory, requiring Splunk Enterprise to contact the disk. Windows counts "soft page faults," so Windows machines exhibit more page faults. Consult Lock percentage and Queues instead for Windows.
Lock percentage	50%+	30%–50%	0–30%	High lock percentage can starve replication and/or cause application calls to be slow, time out, or fail. High lock percentage typically means that heavy write activity is occurring on the node.
Network traffic	N/A	N/A	N/A	Network traffic should be commensurate with system use and application expectations. No default thresholds apply.
Replication latency	>30 seconds	10–30 seconds	0–10 seconds	Replication needs are system dependent. Generally, replica set members should not fall significantly behind the KV captain. Replication latency over 30 seconds can indicate a mounting replication problem.
Primary operations log window	N/A	N/A	N/A	Provided for reference. This is the amount of data, in terms of time, a system saved in the operations log for restoration.
Flushing rate	50%–100%	10%–50%	0–10%	A high flush rate indicates heavy write operations or sluggish system performance.

Troubleshoot this view

The historical panels get data from the _introspection and _internal indexes. Gaps in time in these panels indicate a point at which KV store or Splunk Enterprise was unreachable. If a panel is completely blank or missing data from specific Splunk Enterprise instances, check:

• that you're forwarding your logs to your indexers, and
• the system requirements for platform instrumentation.