Splunk® Enterprise

Monitoring Splunk Enterprise

Acrobat logo Download manual as PDF

Splunk Enterprise version 6.x is no longer supported as of October 23, 2019. See the Splunk Software Support Policy for details. For information about upgrading to a supported version, see How to upgrade Splunk Enterprise.
This documentation does not apply to the most recent version of Splunk. Click here for the latest version.
Acrobat logo Download topic as PDF

Search activity dashboards

This topic is a reference for the Search activity: Instance and Search activity: Deployment dashboards in the Distributed Management Console. See About the Distributed Management Console in this manual.

What do these views show?

Several panels about search activity across your Splunk Enterprise deployment and broken down by instance.

Interpret results in these views

In the Median Resource Usage of Searches panels, note that:

  • Resource usage is aggregated over all searches.
  • Memory usage represents physical memory.
  • In this chart, CPU usage is expressed in percentage of one core, not as system-wide CPU usage. As a result, you are likely to see values >100% here. This is not the case for other examples of CPU usage in the distributed management console.

In the Aggregate Search Runtime panel, note that:

  • For each time bin in the chart, the DMC adds up the runtime of all searches that were running during that time range. Thus, you might see, for example, 1000 seconds of search in 5 minutes. This means that multiple searches were running over the course of those 5 minutes.
  • For the modes historical batch and RT indexed, historical batch can be dispatched only by certain facilities within Splunk Enterprise (the scheduler, for example). RT indexed means indexed real-time.

In the Top 10 Memory-Consuming Searches panel, SID means search ID. If you are looking for information about a saved search, audit.log matches the name of your saved search (savedsearch_name) with its search ID (search_id), user, and time. With the search_id, you can look up that search elsewhere, like in the Splunk platform search logs (see What Splunk logs about itself).

Interpret results in these views

The memory and CPU usage shown here are for searches only. See the resource usage dashboards for all Splunk Enterprise resource usage.

In the Instances by Median CPU Usage panel, CPU can be greater than 100% because of multiple cores.

In the Instances by Median Memory Usage panel, memory is physical.

For the modes historical batch and RT indexed: historical batch can be dispatched only by certain facilities within Splunk Enterprise (the scheduler, for example). RT indexed means indexed real-time.

What to look for in these views

Consider your search concurrency and resource usage compared to your system limits.

For information, see:

A general pattern when looking over all the panels in this dashboards is to look for things that are close to exceeding their limits on your machines.

Troubleshoot these views

The historical panels get data from introspection logs. If a panel is blank or missing information from non-indexers, check:

In the Search Activity: Instance > Search activity panel, the snapshots are taken every ten seconds by default. So if no searches are currently running, or if the searches you run are very short lived, the snapshots panel is blank and says "no results found."

Last modified on 02 August, 2016
Platform alerts
Search usage statistics dashboards

This documentation applies to the following versions of Splunk® Enterprise: 6.3.0, 6.3.1, 6.3.2, 6.3.3, 6.3.4, 6.3.5, 6.3.6, 6.3.7, 6.3.8, 6.3.9, 6.3.10, 6.3.11, 6.3.12, 6.3.13, 6.3.14, 6.4.0, 6.4.1, 6.4.2, 6.4.3, 6.4.4, 6.4.5, 6.4.6, 6.4.7, 6.4.8, 6.4.9, 6.4.10, 6.4.11

Was this documentation topic helpful?

You must be logged into splunk.com in order to post comments. Log in now.

Please try to keep this discussion focused on the content covered in this documentation topic. If you have a more general question about Splunk functionality or are experiencing a difficulty with Splunk, consider posting a question to Splunkbase Answers.

0 out of 1000 Characters