Splunk® Enterprise

Monitoring Splunk Enterprise

Download manual as PDF

Download topic as PDF

Search: Search Activity

This topic is a reference for the Search Activity: Instance and Search Activity: Deployment dashboards in the Monitoring Console. See About the Monitoring Console.

What do these dashboards show?

The search activity dashboards provide an overview of search activity across your Splunk Enterprise deployment as well as more detailed information broken down by instance.

Interpret results in these dashboards

In the Median Resource Usage of Searches panels, note that:

  • Resource usage is aggregated over all searches.
  • Memory usage represents physical memory.
  • In this chart, CPU usage is expressed in percentage of one core, not as system-wide CPU usage. As a result, you are likely to see values >100% here. This is not the case for other examples of CPU usage in the distributed management console.

In the Aggregate Search Runtime panel, note that:

  • For each time bin in the chart, the Monitoring Console adds up the runtime of all searches that were running during that time range. Thus, you might see, for example, 1000 seconds of search in 5 minutes. This means that multiple searches were running over the course of those 5 minutes.
  • For the modes historical batch and RT indexed, historical batch can be dispatched only by certain facilities within Splunk Enterprise (the scheduler, for example). RT indexed means indexed real-time.

In the Top 10 Memory-Consuming Searches panel, SID means search ID. If you are looking for information about a saved search, audit.log matches the name of your saved search (savedsearch_name) with its search ID (search_id), user, and time. With the search_id, you can look up that search elsewhere, like in the Splunk platform search logs (see What Splunk logs about itself).

The memory and CPU usage shown in these dashboards are for searches only. See the resource usage dashboards for all Splunk Enterprise resource usage.

In the Instances by Median CPU Usage panel, CPU can be greater than 100% because of multiple cores.

In the Instances by Median Memory Usage panel, memory is physical.

For the modes historical batch and RT indexed: historical batch can be dispatched only by certain facilities within Splunk Enterprise (the scheduler, for example). RT indexed means indexed real-time.

What to look for in these dashboards

Consider your search concurrency and resource usage compared to your system limits.

For information, see:

A general pattern when looking over all the panels in this dashboards is to look for things that are close to exceeding their limits on your machines.

Troubleshoot these dashboards

The historical panels get data from introspection logs. If a panel is blank or missing information from non-indexers, check:

In the Search Activity: Instance > Search activity panel, the snapshots are taken every ten seconds by default. So if no searches are currently running, or if the searches you run are very short lived, the snapshots panel is blank and says "no results found."

Indexing: Indexer Clustering: Service Activity
Search: Search Usage Statistics

This documentation applies to the following versions of Splunk® Enterprise: 6.5.0, 6.5.1, 6.5.1612 (Splunk Cloud only), 6.5.2, 6.5.3, 6.5.4, 6.5.5, 6.5.6, 6.5.7, 6.5.8, 6.5.9, 6.5.10, 6.6.0, 6.6.1, 6.6.2, 6.6.3, 6.6.4, 6.6.5, 6.6.6, 6.6.7, 6.6.8, 6.6.9, 6.6.10, 6.6.11, 6.6.12, 7.0.0, 7.0.1, 7.0.2, 7.0.3, 7.0.4, 7.0.5, 7.0.6, 7.0.7, 7.0.8, 7.0.9, 7.1.0, 7.1.1, 7.1.2, 7.1.3, 7.1.4, 7.1.5, 7.1.6, 7.1.7, 7.2.0, 7.2.1, 7.2.2, 7.2.3, 7.2.4, 7.2.5, 7.2.6

Was this documentation topic helpful?

Enter your email address, and someone from the documentation team will respond to you:

Please provide your comments here. Ask a question or make a suggestion.

You must be logged into splunk.com in order to post comments. Log in now.

Please try to keep this discussion focused on the content covered in this documentation topic. If you have a more general question about Splunk functionality or are experiencing a difficulty with Splunk, consider posting a question to Splunkbase Answers.

0 out of 1000 Characters