Splunk® Enterprise

Monitoring Splunk Enterprise

Download manual as PDF

This documentation does not apply to the most recent version of Splunk. Click here for the latest version.
Download topic as PDF

Search activity dashboards

This topic is a reference for the Search activity: Instance and Search activity: Deployment dashboards in the Distributed Management Console. See About the Distributed Management Console in this manual.

What do these views show?

Several panels about search activity across your Splunk Enterprise deployment and broken down by instance.

Interpret results in these views

In the Median Resource Usage of Searches panels, note that:

  • Resource usage is aggregated over all searches.
  • Memory usage represents physical memory.
  • In this chart, CPU usage is expressed in percentage of one core, not as system-wide CPU usage. As a result, you are likely to see values >100% here. This is not the case for other examples of CPU usage in the distributed management console.

In the Aggregate Search Runtime panel, note that:

  • For each time bin in the chart, the DMC adds up the runtime of all searches that were running during that time range. Thus, you might see, for example, 1000 seconds of search in 5 minutes. This means that multiple searches were running over the course of those 5 minutes.
  • For the modes historical batch and RT indexed, historical batch can be dispatched only by certain facilities within Splunk Enterprise (the scheduler, for example). RT indexed means indexed real-time.

In the Top 10 Memory-Consuming Searches panel, SID means search ID. If you are looking for information about a saved search, audit.log matches the name of your saved search (savedsearch_name) with its search ID (search_id), user, and time. With the search_id, you can look up that search elsewhere, like in the Splunk platform search logs (see What Splunk logs about itself).

Interpret results in these views

The memory and CPU usage shown here are for searches only. See the resource usage dashboards for all Splunk Enterprise resource usage.

In the Instances by Median CPU Usage panel, CPU can be greater than 100% because of multiple cores.

In the Instances by Median Memory Usage panel, memory is physical.

For the modes historical batch and RT indexed: historical batch can be dispatched only by certain facilities within Splunk Enterprise (the scheduler, for example). RT indexed means indexed real-time.

What to look for in these views

Consider your search concurrency and resource usage compared to your system limits.

For information, see:

A general pattern when looking over all the panels in this dashboards is to look for things that are close to exceeding their limits on your machines.

Troubleshoot these views

The historical panels get data from introspection logs. If a panel is blank or missing information from non-indexers, check:

In the Search Activity: Instance > Search activity panel, the snapshots are taken every ten seconds by default. So if no searches are currently running, or if the searches you run are very short lived, the snapshots panel is blank and says "no results found."

PREVIOUS
Platform alerts
  NEXT
Search usage statistics dashboards

This documentation applies to the following versions of Splunk® Enterprise: 6.3.0, 6.3.1, 6.3.2, 6.3.3, 6.3.4, 6.3.5, 6.3.6, 6.3.7, 6.3.8, 6.3.9, 6.3.10, 6.3.11, 6.3.12, 6.3.13, 6.3.14, 6.4.0, 6.4.1, 6.4.2, 6.4.3, 6.4.4, 6.4.5, 6.4.6, 6.4.7, 6.4.8, 6.4.9, 6.4.10, 6.4.11


Was this documentation topic helpful?

Enter your email address, and someone from the documentation team will respond to you:

Please provide your comments here. Ask a question or make a suggestion.

You must be logged into splunk.com in order to post comments. Log in now.

Please try to keep this discussion focused on the content covered in this documentation topic. If you have a more general question about Splunk functionality or are experiencing a difficulty with Splunk, consider posting a question to Splunkbase Answers.

0 out of 1000 Characters