Resource Usage

This topic is a reference for the Resource Usage dashboards in the Monitoring Console. See About the Monitoring Console.

What do these dashboards show?

There are several Resource Usage dashboards, which you access through the Resource Usage menu. The Resource Usage: Deployment dashboard provides deployment-wide resource information, such as CPU usage, physical memory usage, and disk usage. These panels can be useful for capacity planning.

The other dashboards provide usage information by instance or machine.

Interpret results in these dashboards

About physical memory usage in these dashboards: on Linux, the OS uses free physical memory to cache filesystem resources. But memory for this is loosely bound, and the OS frees it up if a higher priority process needs it. The Monitoring Console reporting cannot discern how much memory is loosely locked up in this way.

The historical data in these dashboards comes from resource_usage.log in the _introspection index. See What data gets logged in the platform instrumentation chapter of the Troubleshooting Manual.

Resource usage: Deployment

The Deployment-Wide Median Disk Usage panel takes into account all partitions in use by each Splunk Enterprise instance.

Resource usage: Machine

This dashboard can be useful for operational post mortems, as well as for capacity planning. See the Capacity Planning Manual for more information.

In the Median CPU Usage panel, 100% means the entire system, however many cores the system has. This is in contrast to the Search Activity dashboards, where 100% means one core.

The disk space in this dashboard refers only to partitions with a Splunk Enterprise instance on them.

Resource usage: Instance

In the two "process class" panels, the value of process class can be splunkd server, search, Splunk Web, index service, scripted input, KVStore, or other.

A process class is an aggregate of processes within one class. For more information about

• splunkd, see Splunk Enterprise architecture and processes in the Installation Manual.
• search, see Get started with Search and Write better searches in the Search Manual.
• splunkweb, see Splunk Enterprise architecture and processes in the Installation Manual.
• scripted input, see Get data from APIs and other remote data interfaces through scripted inputs in the Getting Data In Manual.
• KVStore, see the KV store dashboards dashboard in the Monitoring Console.

The index service consists of housekeeping tasks related to indexing. These tasks run at the end of the indexing pipeline but are asynchronous. These processes run on their own, not through splunkd.

The Disk Usage and Median Disk Usage panels list only the partitions that Splunk Enterprise uses.

What to look out for in these dashboards

The panels of the deployment-wide dashboard group instances by value ranges. In this dashboard, look for outliers: instances that are not like the others.

In all of the resource usage dashboards, look for patterns that appear over time. For example, in the instance and machine dashboards, look for memory usage increasing over time without recovering.

In the Instance dashboard, if a process class that is using a lot of resources turns out to be search, investigate further by going to the Search activity: Instance dashboard.

Troubleshoot these dashboards

The historical panels get data from introspection logs. If a panel is blank or missing information from non-indexers, check:

• that you are forwarding your introspection logs to your indexers, and
• the system requirements for platform instrumentation.