Splunk® Enterprise

Admin Manual

Download manual as PDF

This documentation does not apply to the most recent version of Splunk. Click here for the latest version.
Download topic as PDF

KV store: Deployment

What does this view show?

The KV store: Deployment view in the distributed management console (DMC) provides information aggregated across all KV stores in your Splunk Enterprise deployment. For an instance to be included in this view, it must be set with the server role of KV store. Do this in the DMC Setup page.

This view and the KV store: Instance view track much of the same information. The difference is that this deployment view collects statistics from KV stores and displays the instances grouped by values of those different metrics.

For definitions and context on the individual dashboards and metrics, see "KV store: instance" in this chapter.

Performance Metrics

Deployment Snapshots

Deployment Snapshot Statistics access the /services/server/introspection/kvstore/serverstatus REST endpoint. For each KV store instance in the deployment, the Deployment Snapshots provide the following information:

  • Instance. The Splunk Enterprise instance name.
  • Memory usage
  • Total queued
  • Current connections
  • Page faults per operation
  • Lock (%)
  • Last flush (ms)
  • Network traffic (MB)
  • Up time (in hours). Amount of time the current instance has been running without restart
  • Replication role. The role the instance plays in the replication set. If the instance is not part of a replication set, it returns "N/A."

Where does this view get its data from?

KV store collects data in the _introspection index.

These statistics are broken into the following components:

  • KVStoreServerStats. Information about how the KV store process is performing as a whole. Polled every 27 seconds.
  • KVStoreCollectionStats. Information about collections within the KV store. Polled every 10 minutes.
  • KVStoreReplicaSetStats. Information about replication data across KV store Instances. Polled every 60 seconds.
  • KVProfilingStats. Information about slow operations. Polled every 5 seconds. Only available when profiling is enabled. Note: Enable profile only on development systems or for troubleshooting issues with KV store performance beyond what is available in the default panels. Profiling can negatively affect system performance and so should not be enabled in production environments.

In addition, KV store produces entries in a number of internal logs collected by Splunk Enterprise.

Interpret this view

Panel Critical Warning Normal Interpretation
Page faults per operation 1.3+

Reads require heavy disk I/O, which could indicate a need for more RAM.

0.7–1.3

Reads regularly require disk I/O.

0–0.7

Reads rarely require disk I/O.

Measures how often read requests are not satisfied by what Splunk Enterprise has in memory, requiring Splunk Enterprise to contact the disk.
Lock percentage 50%+ 30%–50% 0–30% High lock percentage can starve replication and/or cause application calls to be slow, time out, or fail. High lock percentage typically means that heavy write activity is occurring on the node.
Network traffic N/A N/A N/A Network traffic should be commensurate with system use and application expectations. No default thresholds apply.
Replication latency >30 seconds 10–30 seconds 0–10 seconds Replication needs are system dependent. Generally, replica set members should not fall significantly behind the KV captain. Replication latency over 30 seconds can indicate a mounting replication problem.
Primary operations log window N/A N/A N/A Provided for reference. This is the amount of data, in terms of time, a system saved in the operations log for restoration.
Flushing rate 50%–100% 10%–50% 0–10% A high flush rate indicates heavy write operations or sluggish system performance.

Troubleshoot this view

The historical panels get data from the _introspection and _internal indexes. Gaps in time in these panels indicate a point at which KV store or Splunk Enterprise was unreachable. If a panel is completely blank or missing data from specific Splunk Enterprise instances, check:

PREVIOUS
KV store: Instance
  NEXT
Licensing

This documentation applies to the following versions of Splunk® Enterprise: 6.2.0, 6.2.1, 6.2.2, 6.2.3, 6.2.4, 6.2.5, 6.2.6, 6.2.7, 6.2.8, 6.2.9, 6.2.10, 6.2.11, 6.2.12, 6.2.13


Was this documentation topic helpful?

Enter your email address, and someone from the documentation team will respond to you:

Please provide your comments here. Ask a question or make a suggestion.

You must be logged into splunk.com in order to post comments. Log in now.

Please try to keep this discussion focused on the content covered in this documentation topic. If you have a more general question about Splunk functionality or are experiencing a difficulty with Splunk, consider posting a question to Splunkbase Answers.

0 out of 1000 Characters