KV store: Deployment
What does this view show?
The KV store: Deployment view in the distributed management console (DMC) provides information aggregated across all KV stores in your Splunk Enterprise deployment. For an instance to be included in this view, it must be set with the server role of KV store. Do this in the DMC Setup page.
This view and the KV store: Instance view track much of the same information. The difference is that this deployment view collects statistics from KV stores and displays the instances grouped by values of those different metrics.
For definitions and context on the individual dashboards and metrics, see "KV store: instance" in this chapter.
Deployment Snapshot Statistics access the
/services/server/introspection/kvstore/serverstatus REST endpoint. For each KV store instance in the deployment, the Deployment Snapshots provide the following information:
- Instance. The Splunk Enterprise instance name.
- Memory usage
- Total queued
- Current connections
- Page faults per operation
- Lock (%)
- Last flush (ms)
- Network traffic (MB)
- Up time (in hours). Amount of time the current instance has been running without restart
- Replication role. The role the instance plays in the replication set. If the instance is not part of a replication set, it returns "N/A."
Where does this view get its data from?
KV store collects data in the _introspection index.
These statistics are broken into the following components:
- KVStoreServerStats. Information about how the KV store process is performing as a whole. Polled every 27 seconds.
- KVStoreCollectionStats. Information about collections within the KV store. Polled every 10 minutes.
- KVStoreReplicaSetStats. Information about replication data across KV store Instances. Polled every 60 seconds.
- KVProfilingStats. Information about slow operations. Polled every 5 seconds. Only available when profiling is enabled. Note: Enable profile only on development systems or for troubleshooting issues with KV store performance beyond what is available in the default panels. Profiling can negatively affect system performance and so should not be enabled in production environments.
In addition, KV store produces entries in a number of internal logs collected by Splunk Enterprise.
Interpret this view
|Page faults per operation||1.3+
Reads require heavy disk I/O, which could indicate a need for more RAM.
Reads regularly require disk I/O.
Reads rarely require disk I/O.
|Measures how often read requests are not satisfied by what Splunk Enterprise has in memory, requiring Splunk Enterprise to contact the disk.|
|Lock percentage||50%+||30%–50%||0–30%||High lock percentage can starve replication and/or cause application calls to be slow, time out, or fail. High lock percentage typically means that heavy write activity is occurring on the node.|
|Network traffic||N/A||N/A||N/A||Network traffic should be commensurate with system use and application expectations. No default thresholds apply.|
|Replication latency||>30 seconds||10–30 seconds||0–10 seconds||Replication needs are system dependent. Generally, replica set members should not fall significantly behind the KV captain. Replication latency over 30 seconds can indicate a mounting replication problem.|
|Primary operations log window||N/A||N/A||N/A||Provided for reference. This is the amount of data, in terms of time, a system saved in the operations log for restoration.|
|Flushing rate||50%–100%||10%–50%||0–10%||A high flush rate indicates heavy write operations or sluggish system performance.|
Troubleshoot this view
The historical panels get data from the _introspection and _internal indexes. Gaps in time in these panels indicate a point at which KV store or Splunk Enterprise was unreachable. If a panel is completely blank or missing data from specific Splunk Enterprise instances, check:
KV store: Instance
This documentation applies to the following versions of Splunk® Enterprise: 6.2.0, 6.2.1, 6.2.2, 6.2.3, 6.2.4, 6.2.5, 6.2.6, 6.2.7, 6.2.8, 6.2.9, 6.2.10, 6.2.11, 6.2.12, 6.2.13, 6.2.14, 6.2.15