Use the Search dashboards

Healthy search loads are critical to the performance of your entire Splunk Cloud Platform environment. Understanding search patterns can help you understand if your search workload is aligned with best practices and optimized for the best performance. The dashboards accessed from the Cloud Monitoring Console > Search tab help you determine if a specific user, search, dashboard, or app is inhibiting your performance. If you encounter an issue, you can then work with users to improve performance.

When searches run for a long time, they may use too much computation and memory, causing an overall slowness of the Splunk instance. This commonly occurs when a few poorly formed searches are taking a large amount of resources. It can also occur if you have a dashboard that is being frequently used by multiple users concurrently. In each of these cases, investigating further can help you to pinpoint the searches that are long-running and determine if you can optimize them.

Because each company's environment is different, it's not easy to set benchmarks for search performance. Generally, the best way to understand your search performance is to compare your historical search times with your current search times to see if there is a change. If search runtimes have slowed, review changes to your environment and new searches to determine if you need to optimize your searches or environment. For example, you may have added a poorly formed search, or you may have added a dashboard that has attracted a lot of traffic.

For more information about optimizing searches, see About search optimization in the Splunk Cloud Platform Search Manual.

A blue progress bar might appear above a panel, indicating that the Splunk platform is still generating data. Wait for the bar to disappear before reviewing the panel.

Do not modify any Cloud Monitoring Console (CMC) dashboard. Changing any of the search criteria, formatting, or layouts may cause inaccurate results and also override the automatic update process.

Analyze search usage statistics

The CMC Search Usage Statistics dashboard provides information about your searches on a per-instance basis. You can split the data by user, host, and source type to better review the results. Analyzing this data helps you answer questions like these:

• Which users run a large number of searches, or long-running searches, or both?
• Which hosts handle the greatest number of searches, and what is the median runtime?
• What are the most heavily used search types in the deployment?

You can then examine searches in more detail to determine if they can be optimized.

See also:

• Write better searches in the Splunk Cloud Platform Search Manual
• Configure the priority of scheduled reports in the Splunk Cloud Platform Reporting Manual

Review the Search Usage Statistics dashboard

This dashboard includes panels of summary, graphical, and tabular data about your search usage statistics. The filter lets you specify a time range and instance values and opt to include or exclude ad hoc searches.

This dashboard contains one panel with a variable in the title: Search Activity by <variable>.

To investigate your panels, go to Cloud Monitoring Console > Search > Search Usage Statistics. Use the following table to understand the dashboard interface.

Panel or Filter	Description
Instances	Choose to view all instances or specify a particular instance.
Time Range	Set the time range for the data display.
Only Ad Hoc Searches	Choose Yes to limit the displayed data to only ad hoc searches. Choose No to view results for both ad hoc and scheduled searches.
Overview	Shows the total number of searches finished, successfully and unsuccessfully completed, and the total search runtime of all searches.
Searches	Provides a graphical representation of the information in the Overview panel. Specify a Split by option to view the related Searches graph by user, host, or search type. Search types displayed grouped in the following general and granular categories: acceleration and summarization: See Overview of summary-based search acceleration. ad hoc: See Ad hoc search. dashboard: Searches that populate a dashboard. Search types in this granular category are ad hoc, acceleration, other, or scheduled. other: Searches that don't fall into the other categories. These searches are generally performed by a Splunk Cloud Platform administrator. realtime: This is a granular category for scheduled searches that are continually running in the background. scheduled: See Scheduled search. This category excludes realtime searches. subsearch: See Subsearch. Search types in this granular category are ad hoc and other.
Search Activity by <variable>	The <variable> is determined by Split by choice. The table lists the following information: Search type and count Median and cumulative runtimes Timestamp of the last search
Cache Activity	Shows your deployment's cache (local storage) activity, aggregated by bucket count or GB. The chart shows the following bucket rates tracked over time: bucket_download: Buckets that have automatically transferred from AWS S3 storage to the cache. Spikes occur when searches need to localize data into the cache and are a part of your deployment's normal execution. Spikes over a longer duration may indicate indexer overload, which affects optimal search and ingest performance. bucket_eviction: Buckets that have been automatically cleared from the cache. bucket_upload: Spikes occur when more data is being transferred to AWS S3, though this does not impact system performance. This metric tracks the following: Buckets that have automatically been added to AWS S3 storage. These are generally hot buckets that are converting to warm buckets. Buckets that already exist in AWS S3 storage but their internal details have changed; this results in an automatic resubmit and re-upload. For example, when a Splunk Cloud Platform administrator uses the delete command, this action affects the bucket's internal details and results in a re-upload of the modified bucket.
Search Details	The table lists the following information: Report name/search string: Shows the report name for saved searches or the search string for ad hoc searches. Search runtime: Enter a value in the Search runtime >= (seconds) field to restrict the results to searches that meet or exceed this runtime. Search start: The initial start time of the search job. Earliest and latest time: The earliest and latest times of the search's time range. Search type User and host names SID: Search identifier

Interpret search usage statistics results

To review searches by user, follow these steps:

1. Change the time frame to widen the scope. For example, set it to the week prior to the current date.
2. Split the search by users so that you can see if there are a few users who are typically running longer searches.
3. Sort by Cumulative Runtime to see which users have the most cumulative search time.
4. Sort by Median Runtime to see which users are running the median longest searches.
5. If the user running the most or longest searches is the system user, you may want to review your applications to make sure that you have optimized them, and that they are providing the expected value. You may discover that some applications are not needed or are not used.

To review long-running searches, follow these steps:

1. Expand the time range to at least 24 hours. Searches are automatically sorted by long-running searches.
2. Set Only Ad Hoc Searches to No if it isn't already. This setting ensures that you see only scheduled searches, which are more likely to be long-running searches than ad hoc searches.
3. Scroll to the Search Details panel where the searches are sorted by search runtime.
4. Select the search name to view more details, and scroll to the bottom of the screen. Two events are displayed. In the second event, you can see the search query.

If you discover a long-running query that runs frequently, you may want to expand the time range to a week or longer to see how commonly this search is run. If it is running frequently, consider optimizing the search.

Be sure to monitor the Cache Activity panel for any spikes in bucket download size and count that have a long duration. Constant and excessive downloading of data into the cache churns the local file system on your indexers, and sustained bucket download spikes may cause performance impacts in searching and ingestion. To remediate these issues, you may need to modify search patterns, alter retention, or upgrade your workload-based or ingestion-based subscription entitlements so more of your searchable data is able to reside in the cache.

Check scheduler activity

The CMC Scheduler Activity dashboard provides information to Splunk Cloud Platform administrators about how search jobs, also known as reports, are scheduled. Use this dashboard to understand the performance of your scheduled activity and determine if it needs any optimization.

See also Configure the priority of scheduled reports in the Splunk Cloud Platform Reporting Manual.

Review the Scheduler Activity dashboard

This dashboard contains a number of panels about your scheduler activity.

Panels are grouped into one of three views, with a fourth view that combines the other three views so you can see all the data concurrently. Filter the results by specifying a time range and opting to include or exclude acceleration searches.

To investigate your panels, go to Cloud Monitoring Console > Search > Scheduler Activity. Use the following table to understand the dashboard interface.

View or Filter	Description
Time Range	Set the time range for the data display.
Select View	Select a specific view, Status, Activity, or Performance, or select All to see a combined view.
Include Acceleration Searches	Acceleration searches are summaries of large datasets, used to help efficiently report on large volumes of data. Select Yes to include summary-based acceleration searches in the displayed results. Select No to include only searches run on the complete dataset in the results.
Status	Contains the following two panels: Skip Ratio (Last Hour) shows the ratio of schedulers that skipped in the last hour.   Average Execution Latency (Last Hour) shows the latency in seconds. The smaller number and the arrow indicate if this is an increase or decrease from the previous reading.
Activity	Contains the following four panels: Scheduler Executions Detail table with a Group by filter. The Total value is affected by the specified Time Range and Include Accelerated Searches filter values. Scheduler Executions bar graph with a Group by drop-down list. Scheduled Report Completions bar graph with a Runtime Aggregation filter. <variable> Concurrency of Scheduled Searches bar graph with Aggregation, Search Mode, and Group by filters. The specified Aggregation value determines the variable. If you select maximum aggregation and historical search mode, The results show the maximum concurrency of completed scheduled searches. The results do not include searches that are currently running or searches that haven't completed within the displayed 5 minute intervals.
Performance	Contains the following two panels: Scheduler Errors and Warnings table. The Total value is affected by the specified Time Range and Include Accelerated Searches filter values. Execution Latency bar graph with Group by and Aggregation filters.

Interpret scheduler activity results

When interpreting your scheduler activity results, note the following:

• Be sure to include and exclude acceleration searches and see how that affects your results.
• Check for any significant spikes or dips in scheduler status, activity, and performance, particularly as compared against historical data. Large spikes and dips generally indicate an issue that you must resolve as soon as possible.
• Choose various time range values when reviewing the Scheduler Errors and Warnings panel. Doing this ensures that you see the errors and warnings reported for that specific time range.

Investigate skipped scheduled searches

The CMC Skipped Scheduled Searches dashboard provides information to Splunk Cloud Platform administrators on skipped searches and search errors. Use this dashboard to identify why the Splunk platform can't process your scheduled searches and take steps to correct the issues.

See also Prioritize concurrently scheduled reports in Splunk Web and Offset scheduled search start times in the Splunk Cloud Platform Reporting Manual. 

Review the Skipped Scheduled Searches dashboard

This dashboard includes six panels of summary, graphical, and tabular data. Filter the results by specifying a time range and opting to include or exclude acceleration searches.

To investigate your panels, go to Cloud Monitoring Console > Search > Skipped Scheduled Searches. Use the following table to understand the dashboard interface.

Panel or Filter	Description
Time Range	Set the time range for the data display.
Include Acceleration Searches	Acceleration searches are summaries of large datasets, used to help efficiently report on large volumes of data. Select Yes to include summary-based acceleration searches in the displayed results. Select No to include only searches run on the complete dataset in the results.
Total Skipped Searches	Shows the total number of skipped searches.
Scheduled Search Skip Ratio	Shows the percentage of your scheduled searches that had to be skipped.
Skipped Scheduled Searches Detail	Shows a table with Group by filter.
Skipped Searches	Shows a bar graph of skipped scheduled searches with Group by filter.
Skipped Scheduled Searches by Name and Reason	Shows a table that lists the following: Report name Skip reason and count Alert actions Total skips
Scheduler Errors and Warnings	Shows a table that lists the following: Error message Count, or number of times the message was issued Percent of Total, indicating what percent of the total this error was issued

Interpret skipped scheduled searches results

If you are skipping searches, it can be indicative of the following possible problems with your search scheduling or query formation:

• You have scheduled too many searches to run at the same time. Alleviate this problem by staggering the scheduled searches.
• You have a search that attempts to run before the previously scheduled search has completed. For example, you schedule Search_A to run every 5 minutes, but the first instance of the search takes 10 minutes to complete. The next time the search is scheduled to run, it is skipped because the first search has not yet completed. Correct this issue by either adjusting the time range (set it to 10 minutes instead of 5) or optimizing your search to improve its performance. See About search optimization in the Splunk Cloud Platform Search Manual.
• You have skipped searches because your users have met the threshold for concurrency limits that you set in your Splunk System Limits. This is expected behavior, but it may also indicate that your users need help in optimizing their searches.

To check for skipped searches, perform these steps:

1. In the Time Range field of Skipped Scheduled Searches, select 24 hours to get a better picture of your searches historically.
2. In the Skipped Scheduled Searches Detail panel, sort by Reason. Frequently, there are a number of skipped searches for the same reason. Note the primary reason or reasons that searches are skipped.
3. Scroll down to see which report is generating the primary issues. Note the report name and determine the following:
• If this is an expected behavior, you don't need to research any further.
• If the skipped searches are unexpected, continue to the next step.
4. Go to Settings > Searches Reports and Alerts.
5. If you know the App associated with the search or report, you can sort by the App. Otherwise, search by the report or search name.
6. Locate the needed search or report and select it to open the Edit Search dialog box.
7. Determine if the problem is with the search formation or the scheduling:
• If you need to troubleshoot the formation of the search, look for wild cards and check whether an index is specified.
• If scheduling is the problem, continue to the next step.
8. Go to Edit > Edit Schedule to review the schedule for the search.
9. Verify that the schedule for the report or search is in line with how long the search takes to complete. For example, if the report runs every hour but it takes 1.5 hours to run the search, the searches are skipped.

Enable the Skipped Scheduled Searches alert

Skipped searches can be indicators of non-optimal performance in your deployment. A high ratio of skipped scheduled searches can indicate the following:

• The number of searches being run exceeds your deployment's capacity.
• The searches being run are taking too long, or using large amounts of memory or CPU.

CMC provides an alert that lets you know when the ratio of skipped scheduled searches exceeds 20% in a 60-minute period. For more information about using this alert, see Use the Alerts panel. For general information about managing alerts, see the Splunk Cloud Platform Alerting Manual.

Analyze expensive searches

The CMC Expensive Searches dashboard provides information to Splunk Cloud Platform administrators on searches that are high consumers of your Splunk Cloud Platform resources. Use this dashboard to determine if these expensive and possibly inefficient searches are worth their cost.

Review the Expensive Searches dashboard

This dashboard provides four panels of data regarding expensive and inefficient searches. Set a time range to filter the results.

To investigate your panels, go to Cloud Monitoring Console > Search > Expensive Searches. Use the following table to understand the dashboard interface.

Panel or Filter	Description
Time Range	Set the time range for the data display.
Maximum Runtime Searches	Shows a line graph of search duration in seconds over time, comparing maximum ad hoc searches against scheduled searches.
Top 20 Most Memory Consuming Searches	Shows a table that lists the following: Splunk platform instance label Provenance Percentage memory used (KB) Search duration and start time and date Search type, mode, and app User name and role
Top 20 Most Expensive Ad Hoc Searches	Shows a table that lists the following: Search time User Time range start and end Search duration and result count Memory usage (KB) Total number of events scanned Search query
Top 20 Most Expensive Scheduled Searches	Shows a table that lists the following: Search time User Scheduled time Status Search duration and result count Memory usage (KB) Saved search name
Potentially Inefficient Searches	Shows a table that lists the following: User Search Processing Language (SPL) Events scanned Search time range in days Search duration Splunk query score: A calculated number based on weighted indicators within an SPL string. A high score indicates a very inefficient search. Potentially inefficient behavior: Shows the indicators within an SPL string that reduce search efficiency.

Interpret expensive searches results

When interpreting your expensive searches results, note the following:

• After you identify the expensive and inefficient searches in your deployment, collaborate with users to improve the queries, using the information in the Write better searches topic in the Splunk Cloud Platform Search Manual.
• Review the score range of your searches using the Splunk Query Score column in the Potentially Inefficient Searches panel, and optimize searches that received a high score as soon as possible.