Use the Search dashboards
Healthy search loads are critical to the performance of your entire Splunk Cloud Platform environment. Understanding search patterns can help you understand if your search workload is aligned with best practices and optimized for the best performance. The dashboards accessed from the Cloud Monitoring Console > Search tab help you determine if a specific user, search, dashboard, or app is inhibiting your performance. If you encounter an issue, you can then work with users to improve performance.
When searches run for a long time, they may use too much computation and memory, causing an overall slowness of the Splunk instance. This commonly occurs when a few poorly formed searches are taking a large amount of resources. It can also occur if you have a dashboard that is being frequently used by multiple users concurrently. In each of these cases, investigating further can help you to pinpoint the searches that are long-running and determine if you can optimize them.
Because each company's environment is different, it's not easy to set benchmarks for search performance. Generally, the best way to understand your search performance is to compare your historical search times with your current search times to see if there is a change. If search runtimes have slowed, review changes to your environment and new searches to determine if you need to optimize your searches or environment. For example, you may have added a poorly formed search, or you may have added a dashboard that has attracted a lot of traffic.
For more information about optimizing searches, see About search optimization in the Splunk Cloud Platform Search Manual.
A blue progress bar might appear above a panel, indicating that the Splunk platform is still generating data. Wait for the bar to disappear before reviewing the panel.
Do not modify any Cloud Monitoring Console (CMC) dashboard. Changing any of the search criteria, formatting, or layouts may cause inaccurate results and also override the automatic update process.
Analyze search usage statistics
The CMC Search Usage Statistics dashboard provides information about your searches on a per-instance basis. You can split the data by user, host, and source type to better review the results. Analyzing this data helps you answer questions like these:
- Which users run a large number of searches, or long-running searches, or both?
- Which hosts handle the greatest number of searches, and what is the median runtime?
- What are the most heavily used search types in the deployment?
You can then examine searches in more detail to determine if they can be optimized.
See also:
- Write better searches in the Splunk Cloud Platform Search Manual
- Configure the priority of scheduled reports in the Splunk Cloud Platform Reporting Manual
Review the Search Usage Statistics dashboard
This dashboard includes panels of summary, graphical, and tabular data about your search usage statistics. The filter lets you specify a time range and instance values and opt to include or exclude ad hoc searches.
This dashboard contains one panel with a variable in the title: Search Activity by <variable>.
To investigate your panels, go to Cloud Monitoring Console > Search > Search Usage Statistics. Use the following table to understand the dashboard interface.
Panel or Filter | Description |
---|---|
Instances | Choose to view all instances or specify a particular instance. |
Time Range | Set the time range for the data display. |
Only Ad Hoc Searches | Choose Yes to limit the displayed data to only ad hoc searches. Choose No to view results for both ad hoc and scheduled searches. |
Overview | Shows the total number of searches finished, successfully and unsuccessfully completed, and the total search runtime of all searches. |
Searches | Provides a graphical representation of the information in the Overview panel. Specify a Split by option to view the related Searches graph by user, host, or search type. Search types displayed grouped in the following general and granular categories:
|
Search Activity by <variable> | The <variable> is determined by Split by choice. The table lists the following information:
|
Cache Activity | Shows your deployment's cache (local storage) activity, aggregated by bucket count or GB.
The chart shows the following bucket rates tracked over time:
|
Search Details | The table lists the following information:
|
Interpret search usage statistics results
To review searches by user, follow these steps:
- Change the time frame to widen the scope. For example, set it to the week prior to the current date.
- Split the search by users so that you can see if there are a few users who are typically running longer searches.
- Sort by Cumulative Runtime to see which users have the most cumulative search time.
- Sort by Median Runtime to see which users are running the median longest searches.
- If the user running the most or longest searches is the system user, you may want to review your applications to make sure that you have optimized them, and that they are providing the expected value. You may discover that some applications are not needed or are not used.
To review long-running searches, follow these steps:
- Expand the time range to at least 24 hours. Searches are automatically sorted by long-running searches.
- Set Only Ad Hoc Searches to No if it isn't already. This setting ensures that you see only scheduled searches, which are more likely to be long-running searches than ad hoc searches.
- Scroll to the Search Details panel where the searches are sorted by search runtime.
- Select the search name to view more details, and scroll to the bottom of the screen. Two events are displayed. In the second event, you can see the search query.
If you discover a long-running query that runs frequently, you may want to expand the time range to a week or longer to see how commonly this search is run. If it is running frequently, consider optimizing the search.
Be sure to monitor the Cache Activity panel for any spikes in bucket download size and count that have a long duration. Constant and excessive downloading of data into the cache churns the local file system on your indexers, and sustained bucket download spikes may cause performance impacts in searching and ingestion. To remediate these issues, you may need to modify search patterns, alter retention, or upgrade your workload-based or ingestion-based subscription entitlements so more of your searchable data is able to reside in the cache.
Check scheduler activity
The CMC Scheduler Activity dashboard provides information to Splunk Cloud Platform administrators about how search jobs, also known as reports, are scheduled. Use this dashboard to understand the performance of your scheduled activity and determine if it needs any optimization.
See also Configure the priority of scheduled reports in the Splunk Cloud Platform Reporting Manual.
Review the Scheduler Activity dashboard
This dashboard contains a number of panels about your scheduler activity.
Panels are grouped into one of three views, with a fourth view that combines the other three views so you can see all the data concurrently. Filter the results by specifying a time range and opting to include or exclude acceleration searches.
To investigate your panels, go to Cloud Monitoring Console > Search > Scheduler Activity. Use the following table to understand the dashboard interface.
View or Filter | Description |
---|---|
Time Range | Set the time range for the data display. |
Select View | Select a specific view, Status, Activity, or Performance, or select All to see a combined view. |
Include Acceleration Searches | Acceleration searches are summaries of large datasets, used to help efficiently report on large volumes of data.
Select Yes to include summary-based acceleration searches in the displayed results. Select No to include only searches run on the complete dataset in the results. |
Status | Contains the following two panels:
The smaller number and the arrow indicate if this is an increase or decrease from the previous reading. |
Activity | Contains the following four panels:
|
Performance | Contains the following two panels:
|
Interpret scheduler activity results
When interpreting your scheduler activity results, note the following:
- Be sure to include and exclude acceleration searches and see how that affects your results.
- Check for any significant spikes or dips in scheduler status, activity, and performance, particularly as compared against historical data. Large spikes and dips generally indicate an issue that you must resolve as soon as possible.
- Choose various time range values when reviewing the Scheduler Errors and Warnings panel. Doing this ensures that you see the errors and warnings reported for that specific time range.
Investigate skipped scheduled searches
The CMC Skipped Scheduled Searches dashboard provides information to Splunk Cloud Platform administrators on skipped searches and search errors. Use this dashboard to identify why the Splunk platform can't process your scheduled searches and take steps to correct the issues.
See also Prioritize concurrently scheduled reports in Splunk Web and Offset scheduled search start times in the Splunk Cloud Platform Reporting Manual.
Review the Skipped Scheduled Searches dashboard
This dashboard includes six panels of summary, graphical, and tabular data. Filter the results by specifying a time range and opting to include or exclude acceleration searches.
To investigate your panels, go to Cloud Monitoring Console > Search > Skipped Scheduled Searches. Use the following table to understand the dashboard interface.
Panel or Filter | Description |
---|---|
Time Range | Set the time range for the data display. |
Include Acceleration Searches | Acceleration searches are summaries of large datasets, used to help efficiently report on large volumes of data.
Select Yes to include summary-based acceleration searches in the displayed results. Select No to include only searches run on the complete dataset in the results. |
Total Skipped Searches | Shows the total number of skipped searches. |
Scheduled Search Skip Ratio | Shows the percentage of your scheduled searches that had to be skipped. |
Skipped Scheduled Searches Detail | Shows a table with Group by filter. |
Skipped Searches | Shows a bar graph of skipped scheduled searches with Group by filter. |
Skipped Scheduled Searches by Name and Reason | Shows a table that lists the following:
|
Scheduler Errors and Warnings | Shows a table that lists the following:
|
Interpret skipped scheduled searches results
If you are skipping searches, it can be indicative of the following possible problems with your search scheduling or query formation:
- You have scheduled too many searches to run at the same time. Alleviate this problem by staggering the scheduled searches.
- You have a search that attempts to run before the previously scheduled search has completed. For example, you schedule Search_A to run every 5 minutes, but the first instance of the search takes 10 minutes to complete. The next time the search is scheduled to run, it is skipped because the first search has not yet completed. Correct this issue by either adjusting the time range (set it to 10 minutes instead of 5) or optimizing your search to improve its performance. See About search optimization in the Splunk Cloud Platform Search Manual.
- You have skipped searches because your users have met the threshold for concurrency limits that you set in your Splunk System Limits. This is expected behavior, but it may also indicate that your users need help in optimizing their searches.
To check for skipped searches, perform these steps:
- In the Time Range field of Skipped Scheduled Searches, select 24 hours to get a better picture of your searches historically.
- In the Skipped Scheduled Searches Detail panel, sort by Reason. Frequently, there are a number of skipped searches for the same reason. Note the primary reason or reasons that searches are skipped.
- Scroll down to see which report is generating the primary issues. Note the report name and determine the following:
- If this is an expected behavior, you don't need to research any further.
- If the skipped searches are unexpected, continue to the next step.
- Go to Settings > Searches Reports and Alerts.
- If you know the App associated with the search or report, you can sort by the App. Otherwise, search by the report or search name.
- Locate the needed search or report and select it to open the Edit Search dialog box.
- Determine if the problem is with the search formation or the scheduling:
- If you need to troubleshoot the formation of the search, look for wild cards and check whether an index is specified.
- If scheduling is the problem, continue to the next step.
- Go to Edit > Edit Schedule to review the schedule for the search.
- Verify that the schedule for the report or search is in line with how long the search takes to complete. For example, if the report runs every hour but it takes 1.5 hours to run the search, the searches are skipped.
Enable the Skipped Scheduled Searches alert
Skipped searches can be indicators of non-optimal performance in your deployment. A high ratio of skipped scheduled searches can indicate the following:
- The number of searches being run exceeds your deployment's capacity.
- The searches being run are taking too long, or using large amounts of memory or CPU.
CMC provides an alert that lets you know when the ratio of skipped scheduled searches exceeds 20% in a 60-minute period. For more information about using this alert, see Use the Alerts panel. For general information about managing alerts, see the Splunk Cloud Platform Alerting Manual.
Analyze expensive searches
The CMC Expensive Searches dashboard provides information to Splunk Cloud Platform administrators on searches that are high consumers of your Splunk Cloud Platform resources. Use this dashboard to determine if these expensive and possibly inefficient searches are worth their cost.
Review the Expensive Searches dashboard
This dashboard provides four panels of data regarding expensive and inefficient searches. Set a time range to filter the results.
To investigate your panels, go to Cloud Monitoring Console > Search > Expensive Searches. Use the following table to understand the dashboard interface.
Panel or Filter | Description |
---|---|
Time Range | Set the time range for the data display. |
Maximum Runtime Searches | Shows a line graph of search duration in seconds over time, comparing maximum ad hoc searches against scheduled searches. |
Top 20 Most Memory Consuming Searches | Shows a table that lists the following:
|
Top 20 Most Expensive Ad Hoc Searches | Shows a table that lists the following:
|
Top 20 Most Expensive Scheduled Searches | Shows a table that lists the following:
|
Potentially Inefficient Searches | Shows a table that lists the following:
|
Interpret expensive searches results
When interpreting your expensive searches results, note the following:
- After you identify the expensive and inefficient searches in your deployment, collaborate with users to improve the queries, using the information in the Write better searches topic in the Splunk Cloud Platform Search Manual.
- Review the score range of your searches using the Splunk Query Score column in the Potentially Inefficient Searches panel, and optimize searches that received a high score as soon as possible.
Use the Indexing dashboards | Use the Usage dashboards |
This documentation applies to the following versions of Splunk Cloud Platform™: 8.2.2112, 8.2.2201, 8.2.2202, 8.2.2203, 9.0.2205, 9.0.2208, 9.0.2209, 9.0.2303, 9.0.2305, 9.1.2308, 9.1.2312, 9.2.2403 (latest FedRAMP release), 9.2.2406
Feedback submitted, thanks!