Splunk® Enterprise

Monitoring Splunk Enterprise

Acrobat logo Download manual as PDF


Splunk Enterprise version 8.2 is no longer supported as of September 30, 2023. See the Splunk Software Support Policy for details. For information about upgrading to a supported version, see How to upgrade Splunk Enterprise.
Acrobat logo Download topic as PDF

Resource Usage: CPU Usage

This topic is a reference for the Resource Usage: CPU Usage dashboards in the Monitoring Console. See About the Monitoring Console in this manual.

These dashboards consolidate existing CPU usage views from other Monitoring Console pages to provide a central location for monitoring the CPU resource consumption of your Splunk Enterprise Deployment.

CPU Usage: Deployment

The CPU Usage: Deployment dashboard shows physical CPU and virtual CPU (vCPU) usage across your distributed deployment. To learn what qualifies as a vCPU, see How vCPU is calculated for infrastructure licensing in the Admin Manual.

To view CPU usage for a specific server role, select the server role from the Role menu.

Snapshot overview

The snapshot view includes the following panels that provide an overview of CPU resources:

  • Effective CPU: This panel shows the average CPU usage (%) across all instances in your deployment, as well as the total count of allocated vCPUs.
  • Search Heads: This panel shows the CPU usage and vCPU count across all instances running as search heads.
  • Indexers: This panel shows CPU usage and vCPU count across all instances running as indexers.
  • CPU per Primary Server Role: This panel shows average CPU usage (% ) and vCPU count based on server role. Use this panel to evaluate the current CPU resource consumption of different server roles. The screen image shows four snapshot view panels that provide an overview of current CPU and vCPU usage across a distributed deployment.

CPU Usage by Instance

The CPU Usage by Instance panel shows the number of CPU cores (CPU/vCPU), CPU Usage (%), and Load Average for each instance in a distributed deployment. CPU Usage (%) and Load Average columns are color-coded based on the CPU status for each instance, as follows: 0-60 is green, 60-80 is yellow, 80-100 is orange, and 100+ is red.

You can use this panel to track the status of CPU usage across multiple instances, and to maintain awareness of possible CPU overload conditions that can hurt performance. Click on any instance in the list to open the CPU Usage: Instance dashboard, where you can view detailed CPU usage information for that specific instance.

CPU usage by instance.png

CPU Usage by Primary Server Role over Splunk Process Type

The CPU usage by Primary Server Role over Splunk Process Type panel shows CPU usage of various Splunk process types for each Splunk server role, including search head, indexer, manager node, and so on. Process types include splunkd, search, index service, Splunk Web, KVStore, scripted inputs, and more.

Use this panel to track the amount of CPU resources that different Splunk process types are consuming on a particular Splunk server role.

Click the "Scale x-axis to actual size" check box to maximize the horizontal view.

CPU usage by process type.png

Deployment-wide Median CPU Usage

The Deployment-wide Median CPU Usage panel shows the total CPU usage % over time across your entire deployment.

You can use this panel to identify trends in overall CPU resource consumption, such as increasing CPU usage, which might indicate an issue with your system that requires investigation.

Median CPU Usage

The Median CPU Usage panel shows CPU % usage over time for individual instances in your deployment.

You can use this panel to identify trends in CPU resource consumption that might indicate an issue with a particular instance.

Median CPU Usage by Process Class

The Median CPU Usage by Process Class panel shows CPU resource usage for each Splunk process class. A "process class" is an aggregate of processes within a single class.

You can use this panel to identify spikes in CPU usage by a particular process, which might indicate an underlying issue. For example, high CPU % usage by scripted inputs might indicate an issue with data ingestion.

CPU usage by process class.png

Median CPU usage of Searches

The Median CPU usage of Searches panel lets you monitor CPU usage based on variety of search characteristics, including search type (ad hoc, scheduled)search mode (real time, historical), app context, user, role, search name, and so on.

Use this panel to identify spikes in CPU usage that can indicate various issues with search performance. For example, high CPU usage by scheduled searches can suggest issues with search concurrency.

Median Historical CPU Usage

The Median Historical CPU Usage panel lets you monitor CPU/vCPU usage on a per instance basis over an extended historical time range.

You can use this panel to evaluate longer-term CPU/vCPU usage trends and requirements in the context of capacity planning or license planning.

Median historical cpu usage.png

The CPU usage metrics data that populates this panel is stored in the _metrics_rollup index. The _metrics_rollup index retains this data for two years by default, as specified in indexes.conf. For information on data retention settings, see indexes.conf.spec.

To view data in this panel, you must enable the metric rollup policy specified in the [index:_metrics] stanza in metric_rollups.conf, as follows:

  1. Edit $SPLUNK_HOME/etc/apps/splunk_monitoring_console/local/metric_rollups.conf.
  2. Set disabled = false to enable the metric rollup policy in in the [index:_metrics] stanza, as shown:
    [index:_metrics]
    defaultAggregation = avg
    dimensionListType = excluded
    metricList = spl.intr.resource_usage.Hostwide.data.cpu_system_pct,\
    spl.intr.resource_usage.Hostwide.data.cpu_user_pct
    metricListType = included
    rollup.0.rollupIndex = _metrics_rollup
    rollup.0.span = 1d
    disabled = false
    

    The above metric rollup policy requires that the _metrics_rollup index be present. This index exists by default and is used exclusively by the CPU usage dashboard.

For more information on configuring metric rollup policies in metric_rollups.conf, see Manage metric rollup policies with configuration files in Metrics.

CPU Usage: Instance

The CPU Usage: Instance dashboard lets you monitor physical CPU and vCPU usage of individual instances.

To view CPU usage for a specific instance, select the instance from the Instance menu.

The CPU Usage panel shows the number of physical and virtual CPUs, as well as CPU Usage and Load Average for the selected instance.

The CPU Usage over Splunk Process Type panel shows CPU usage of different Splunk process types for the selected instance.

The Median CPU Usage by Process Class panel show CPU usage for each Splunk Process class on the selected instance.

The Median CPU Usage of Searches panel lets you monitor CPU usage based on variety of search characteristics, including search type (ad hoc, scheduled)search mode (real time, historical), app context, user, role, search name, and so on.

The Median Historical CPU Usage panel lets you view CPU usage of a particular instance over an extended historical time range. Use this panel to evaluate long-term CPU usage trends for capacity planning or license planning purposes.

Troubleshoot these dashboards

CPU/vCPU dashboards get CPU usage information from the _introspection index and /server/status/resource-usage/hostwide endpoint. If a panel is missing data, check the following:

For endpoint information, see /server/status/resource-usage/hostwide in the 'REST API Reference Manual.

Last modified on 19 March, 2022
PREVIOUS
Resource Usage
  NEXT
Forwarders

This documentation applies to the following versions of Splunk® Enterprise: 8.1.0, 8.1.1, 8.1.2, 8.1.3, 8.1.4, 8.1.5, 8.1.6, 8.1.7, 8.1.8, 8.1.9, 8.1.10, 8.1.11, 8.1.12, 8.1.13, 8.1.14, 8.2.0, 8.2.1, 8.2.2, 8.2.3, 8.2.4, 8.2.5, 8.2.6, 8.2.7, 8.2.8, 8.2.9, 8.2.10, 8.2.11, 8.2.12, 9.0.0, 9.0.1, 9.0.2, 9.0.3, 9.0.4, 9.0.5, 9.0.6, 9.0.7, 9.0.8, 9.0.9, 9.1.0, 9.1.1, 9.1.2, 9.1.3, 9.1.4, 9.2.0, 9.2.1


Was this documentation topic helpful?


You must be logged into splunk.com in order to post comments. Log in now.

Please try to keep this discussion focused on the content covered in this documentation topic. If you have a more general question about Splunk functionality or are experiencing a difficulty with Splunk, consider posting a question to Splunkbase Answers.

0 out of 1000 Characters