Search: Search Activity

This topic is a reference for the Search Activity: Instance and Search Activity: Deployment dashboards in the Monitoring Console. See About the Monitoring Console.

What do these dashboards show?

The search activity dashboards provide an overview of search activity across your Splunk Enterprise deployment as well as more detailed information broken down by instance.

Interpret results in these dashboards

In the Median Resource Usage of Searches panels, note that:

• Resource usage is aggregated over all searches.
• Memory usage represents physical memory.
• In this chart, CPU usage is expressed in percentage of one core, not as system-wide CPU usage. As a result, you are likely to see values >100% here. This is not the case for other examples of CPU usage in the distributed management console.

In the Aggregate Search Runtime panel, note that:

• For each time bin in the chart, the Monitoring Console adds up the runtime of all searches that were running during that time range. Thus, you might see, for example, 1000 seconds of search in 5 minutes. This means that multiple searches were running over the course of those 5 minutes.
• For the modes historical batch and RT indexed, historical batch can be dispatched only by certain facilities within Splunk Enterprise (the scheduler, for example). RT indexed means indexed real-time.

In the Top 10 Memory-Consuming Searches panel, SID means search ID. If you are looking for information about a saved search, audit.log matches the name of your saved search (savedsearch_name) with its search ID (search_id), user, and time. With the search_id, you can look up that search elsewhere, like in the Splunk platform search logs (see What Splunk logs about itself).

The memory and CPU usage shown in these dashboards are for searches only. See the resource usage dashboards for all Splunk Enterprise resource usage.

In the Instances by Median CPU Usage panel, CPU can be greater than 100% because of multiple cores.

In the Instances by Median Memory Usage panel, memory is physical.

For the modes historical batch and RT indexed: historical batch can be dispatched only by certain facilities within Splunk Enterprise (the scheduler, for example). RT indexed means indexed real-time.

What to look for in these dashboards

Consider your search concurrency and resource usage compared to your system limits.

For information, see:

• Write better searches in the Search Manual.
• Get started with Search in the Search Manual.
• Configure the priority of scheduled reports in the Reporting Manual.
• Overview of summary-based search and pivot acceleration in the Knowledge Manager Manual.
• How concurrent users and searches impact performance in the Capacity Planning Manual.

A general pattern when looking over all the panels in this dashboards is to look for things that are close to exceeding their limits on your machines.

Troubleshoot these dashboards

The historical panels get data from introspection logs. If a panel is blank or missing information from non-indexers, check:

• that you are forwarding your introspection logs to your indexers, and
• the system requirements for platform instrumentation.

In the Search Activity: Instance > Search activity panel, the snapshots are taken every ten seconds by default. So if no searches are currently running, or if the searches you run are very short lived, the snapshots panel is blank and says "no results found."