Configure the priority of scheduled reports

The Splunk software's report scheduler manages scheduled reports and alerts. Based on your system configuration, the report scheduler sets a limit on the number of reports that can run concurrently. When more reports are scheduled to run than can run concurrently, it prioritizes the excess reports and tries to run them in priority order.

The report scheduler sets limits to ensure that few reports get skipped disproportionally more than others. Reports can be skipped when slow-to-complete scheduled reports crowd out quick-to-complete reports and cause them to miss their scheduled run times.

Why read this topic?

Read this topic when you:

• Have several reports scheduled to run concurrently, and you discover that some of them are consistently skipping their scheduled runs.
• Want to have a detailed understanding of how the scheduling of concurrent reports is managed.
• Need to disable the report scheduler.
• Use Splunk Enterprise and have access to modify configuration files.

For information about how to manually set up a scheduled report, see "Schedule reports" in this manual.

The priority order of different types of searches

The report scheduler determines the run priority for your scheduled reports and alerts. This is important when your resources are limited and only a few reports can run concurrently. The report scheduler runs the reports that it can run at the current time and arranges for the rest to run in a priority order.

The table shows the high level priority in which different types of searches and reports are run.

Priority	Search or Report Type	Description
First priority	Ad hoc historical searches	Historical searches that you run manually always run first. If you start several ad hoc searches at the same time that a number of reports are scheduled to run, some of those reports might get bumped off their schedules to make room for the ad hoc search jobs. For example, your system can run only three scheduled reports concurrently. It is 12 P.M., and you have three scheduled reports that run hourly. A few seconds before noon, you manually run three historical searches. The report scheduler lets your ad hoc searches run concurrently and then queues up the three scheduled reports to run over their scheduled time periods as the ad hoc searches finish.
Second priority	Manually scheduled reports and alerts with real-time scheduling	Reports that you schedule manually use the real-time scheduling mode by default. See the information about real-time scheduling and continuous scheduling in this topic. Within this search and report category, additional priority rules are applied to reduce instances where runs of manually scheduled reports and alerts are skipped. See "How the report scheduler prevents skipped report runs," in this topic.
Third priority	Manually scheduled reports with continuous scheduling	The continuous scheduling mode is used by scheduled reports that populate summary indexes as well as other scheduled reports that cannot have a gap in the collection of report data. See the information about real-time scheduling and continuous scheduling in this topic.
Last priority	Automatically scheduled reports	The processes behind report acceleration and data model acceleration use scheduled reports. These reports are scheduled when you set up acceleration for a report or data model. They generate and update acceleration summaries. If these "auto-summarization" reports have a scheduling conflict with ad hoc searches, alerts, and manually scheduled reports, the auto-summarization reports always runs last. This means that you might have situations where a summary is not being created or updated because other reports with higher priority are running. See "Limits for concurrent scheduled reports," in this topic.

How the report scheduler determines the concurrent scheduled report limit

If you have Splunk Cloud, your deployment is optimized for the performance of searches and reports, and you can work with Splunk Support as required to ensure you are getting desired results. In Splunk Enterprise, you can optimize performance manually, and to do so requires an understanding of the scheduling logic described in this section.

The Splunk report scheduler limits the number of scheduled reports that can be run concurrently. Without these limits, your report performance suffers. To determine what this limit should be, the report scheduler starts with the system-wide limit for concurrent ad hoc historical searches.

See "Accomodate many simultaneous searches" in the Capacity Planning Manual for information about the impact of concurrent searches on search performance.

Caution: If you have Splunk Enterprise, do not change limits.conf settings unless you know what you are doing.

Splunk software determines its system-wide concurrent historical search limit through a calculation where the primary variable is the number of CPUs in the deployment. The calculation includes two parameters that are defined in "limits.conf" in the Admin Manual:

• max_searches_per_cpu is the maximum number of concurrent historical searches allowed per CPU. Defaults to 1.
• base_max_searches is a baseline constant to add to the maximum number of searches, computed as a multiplier of the CPUs. Defaults to 6.

The calculation is as follows:

System-wide maximum number of concurrent historical searches = (max_searches_per_cpu x number of CPUs) + base_max_searches

If your system has one CPU, you can concurrently run a maximum of 7 historical searches ((1 x 1) + 6 = 7).

The report scheduler feeds the system-wide limit for historical searches into another limits.conf parameter: max_searches_perc. The max_searches_perc parameter sets the top number of scheduled reports that can be run concurrently by the report scheduler to 50% of the system-wide limit for concurrent historical searches.

If your system has one CPU, the report scheduler can safely run only three scheduled reports at a time (50% of 7 = 3.5).

The max_searches_perc parameter can be set up so that it allows more or less concurrent scheduled reports depending on the time of day or week. See "Have maximum concurrent scheduled report limits vary by time" next.

Have maximum concurrent scheduled report limits vary by time

You can configure max_searches_perc.n parameter in limits.conf to have regular periods when more or fewer concurrent scheduled reports are allowed. You can use max_searches_perc.n and max_searches_perc.n.when to set up alternative percentages for specific cron periods.

For example:

# The default limit, used when the periods defined below are not in effect.
max_searches_perc = 50 
 
# Increase the value between midnight-5AM.
max_searches_perc.0 = 75
max_searches_perc.0.when = * 0-5 * * *
 
# More specifically, increase it even more on weekends.
max_searches_perc.1 = 90
max_searches_perc.1.when = * 0-5 * * 0,6

The n notation can be any nonnegative number up to 255. The scheduler looks in reverse n order for the first match. If no n settings are provided or if no when matches the current time, the value falls back to the default value for max_searches_perc.

Limits for concurrent scheduled searches that populate acceleration summaries

When you create report and data model acceleration summaries, your Splunk software automatically generates and runs scheduled searches to populate those summaries. These autogenerated scheduled searches also have concurrent search limitations. These limitations are governed by the auto_summary_perc setting.

By default, auto_summary_perc sets a maximum concurrent search value that is 50% of the maximum number of concurrent scheduled searches in your Splunk implementation. The maximum number of concurrent scheduled searches is governed by max_searches_perc, which is itself a percentage of the system-wide limit of historical searches. The system-wide concurrent search limit for historical searches is set by max_searches_per_cpu x number_of_cpus + base_max_searches.

So, what does this mean? Let's say that your max_searches_perc and auto_summary_perc are both set to 50%. If your system-wide concurrent search limit for historical searches is 10, the limit for concurrent scheduled searches (max_searches_perc) would be 50% of that value, or 5. And your limit for concurrent scheduled summary-populating searches (auto_summary_perc) would be 50% of that, or 2.5. This number is rounded down because of integer arithmetic to 2.

In other words, if your system-wide concurrent search limit for historical searches is 10, your concurrent search limit for acceleration-summary-populating scheduled searches is 2.

Use the auto_summary_perc.n parameter to set time intervals when more or less concurrent automatically scheduled reports are allowed. It follows the same logic as the max_searches_perc.n parameter described in "Have maximum concurrent scheduled report limits vary by time."

See "Manage report acceleration" and "Accelerate data models" in the Knowledge Manager Manual.

Set relative concurrency limits for scheduled searches in Splunk Web

You can set relative concurrency limits for scheduled searches to meet your specific search requirements. For example, if you primarily run scheduled searches, you can allocate a larger percentage of searches to scheduled searches. Or, if you primarily use data model acceleration searches, you can allocate a larger percentage of searches to summarization searches.

You can set the the following relative concurrency limits in Splunk Web:

• Scheduled searches: Scheduled searches are searches that you schedule using the Search Scheduler or that are created as part of report acceleration or data model acceleration. You can modify this value to increase or decrease the number of scheduled searches you can run, relative to the system-wide limit for historical searches. For example, if your concurrent limit of 50% results in 19 scheduled searches, and you change the limit to 75%, the maximum number of concurrent scheduled searches changes to 27.

• Summarization searches: Summarization searches are searches that are generated for report acceleration or data model acceleration, For this limit, you set the value as a percentage of the resources allocated for scheduled searches. For example, if you set a scheduled search concurrency limit of 50%, and this results in a limit of 18 concurrent searches, and you set the summarization searches limit to 50%, then summarization searches are allocated roughly 25% of total search resources or roughly 9 summarization searches.

You must have the edit_search_concurrency_all or edit_search_concurrency_scheduled capability to configure these settings.

To set relative concurrency limits in Splunk Web:

1. In Splunk Web, click Settings > Server settings > Search Preferences.
2. In the Relative concurrency limits for scheduled searches select a percentage value for the concurrency limit for scheduled searches. This includes user-scheduled searches and summarization searches.
3. In the Relative concurrency limits for summarization searches select a percentage value for the concurrency limit for summarization searches. This value represents a percentage of the total allocated resources for scheduled searches.
4. Click Save.

Modifying the relative concurrency limit for scheduled searches in the UI automatically updates the max_search_perc value in limits.conf. See How the report scheduler determines the concurrent scheduled report limit.

Likewise, modifying the relative concurrency limit for summarization searches in the UI automatically updates the auto_summary_perc value in limits.conf. See Limits for concurrent scheduled reports that populate acceleration summaries.

Alternatively, you can modify relative concurrency limits for scheduled searches and summarization searches by editing limits.conf or via the search/concurrency-settings/scheduler endpoint.

In a search head cluster, relative concurrency limits that you set in the UI automatically replicate to all cluster members.

Change the report scheduling mode to prioritize completeness of data over recency of data

Each manually scheduled search or report has a scheduling mode. The scheduling mode of a scheduled report determines how the report handles situations where it cannot run at its scheduled run time because of a concurrent search conflict. For example, scheduled reports can skip scheduled search job runs when other long-running report jobs are slow to finish, or when multiple reports with real-time scheduling are set to run at the same time, and their number is higher than the concurrent scheduled report limit

There are two report scheduling modes: real-time scheduling and continuous scheduling. All scheduled reports default to real-time scheduling when they are first created. The report scheduler always prioritizes scheduled reports with the real-time scheduling mode over scheduled reports that have the continuous scheduling mode.

The real-time scheduling mode is not related to real-time search. The real-time scheduling mode can apply to any kind of scheduled search, including scheduled searches that run over historical time ranges.

Scheduling mode	Description	Default for scheduled reports?
real-time scheduling	Real-time scheduling prioritizes current data over complete data. If a scheduled report cannot run on its schedule for some reason, it skips that run and moves on to the next scheduled run. This ensures that any successful run of the report contains up-to-date data, because it is guaranteed to run on its schedule.	Yes
continuous scheduling	Continuous scheduling prioritizes completeness of data over recency of data. The continuous scheduling mode ensures that each scheduled run or a report is eventually performed. If a scheduled report with the continuous scheduling mode cannot run now, it runs later. Apply the continuous scheduling mode to a scheduled report if it is most important that the scheduled report not have any gaps in its collected data.	No

The report scheduler gives reports with real-time scheduling priority over those with continuous scheduling.

Scheduling mode example

How is the real-time scheduling mode different from the continuous scheduling mode, and under what conditions would you prefer one option over the other?

In this example, you have two scheduled reports that are called A and B:

• Report A runs every minute and takes 30 seconds to finish.
• Report B runs every 5 minutes and takes 2 minutes to finish.

Your Splunk Enterprise configuration enables the report scheduler to run only one report at a time.

Both reports are scheduled to run at 1:05 P.M.

Time	Scheduler action
1:05:00 P.M.	The scheduler runs report A for the 1:04 to 1:05 period and schedules it to run again at 1:06 P.M. It is 1:05:30 P.M. when report A finishes.
1:05:30 P.M.	The scheduler runs report B. Because it takes 2 minutes to run, report B will not finish until 1:07:30.
1:06:00 P.M.	The scheduler wakes up and attempts to run report A, but it cannot run because report B is still in process.
1:06:59 P.M.	The scheduler continues to attempt to run report A until 1:06:59. At this point what happens next depends on whether report A is using real-time or continuous scheduling (see below).

If report A is configured to have:

• Real-time scheduling, the scheduler skips the 1:05-1:06 run of the report and schedules the next run of report A for 1:07:00 P.M. (for the 1:06 to 1:07 period). The new report run time is based on the current scheduled run time (1:06:00 P.M.).
• Continuous scheduling, the scheduler does not advance the schedule and attempts to run the report for the 1:05 to 1:06 P.M. period indefinitely. Whatever the eventual report run time is, the next time period that report A covers is 1:06 to 1:07 P.M.

Configure the scheduling mode for individual reports

In Splunk Web you can use the Advanced Edit page to change the scheduling mode for an individual report.

1. Select Settings > Searches, Reports, and Alerts.
2. For the scheduled report that needs a scheduling mode change, select Edit > Advanced Edit.
   Scheduled reports have the time of their next scheduled search job in the Next Scheduled Time column.
3. On the Advanced Edit page, scroll down to realtime_schedule. Set realtime_schedule to false to give the report the continuous scheduling mode. Set realtime_schedule to true to give the report the real-time scheduling mode.
4. Click Save to save your change to the report.

When scheduled reports cannot have missing results

When the Splunk platform is shut down, scheduled reports can miss scheduled runtimes. This can be a big problem for scheduled reports that cannot afford to have missing results. For example, summary indexes are not trustworthy when they are built by scheduled reports that have skipped searches.

There are two ways you can set up a scheduled report so that it won't have missing results: the continuous scheduling mode and durable search processing.

The continuous scheduling mode tries to ensure that every scheduled run a report is eventually performed. This is why, when you enable a scheduled report to be a report that builds a summary index, the search scheduler automatically changes the scheduling mode of the report to continuous scheduling. See Use summary indexing for increased reporting efficiency in the Knowledge Manager Manual.

However, the continuous scheduling mode cannot help with data outages that occur when search heads and search peers go offline for periods of time. The continuous scheduling mode also cannot help you with scheduled search runs that return incomplete result sets. To guard your scheduled reports against such situations, set them up with durable search processing. Durable search processing uses backfill searches to replace the data lost by scheduled search jobs that are skipped or which return incomplete results. Durable search processing ensures that summary indexing searches will not lose data even when data outages last for days, weeks, or months.

Durable search processing is designed to work with the real-time scheduling mode. If you enable durable search processing for a summary indexing report, the search scheduler automatically changes the report's scheduling mode back to real-time scheduling.

See Make scheduled reports durable to prevent event loss in the Reporting Manual.

How the scheduler prevents skipped report runs

The real-time scheduling mode can cause reports to be skipped. To mitigate this, the report scheduler applies a set of rules over the scheduling modes. These rules reduce the number of skipped report runs.

The report scheduler applies three of these rules to all scheduled reports by default. You can manually apply the fourth rule, a scheduling window, to selected reports that do not need to run at their scheduled start times. This gives the report scheduler more flexibility to schedule reports that require more precision.

Rule	Description	Result	Default for all scheduled reports?
Apply a scheduled report priority score	The report scheduler gives each scheduled report a priority score based on its average runtime. It prioritizes lower scores (lower average runtimes) over higher scores (higher average runtimes).	When two or more reports are scheduled to run at the same time, the report scheduler runs the report with the fastest average completion time first. Quick-completing reports are not skipped as often, but slow-completing reports get skipped more.	Yes.
Prioritize real-time scheduling over continuous scheduling	The report scheduler makes sure that scheduled reports with real-time scheduling have better priority scores than scheduled reports that use continuous scheduling.	Reports with real-time scheduling compete for priority only with other scheduled reports with real-time scheduling. Reports with continuous scheduling always get run after reports with real-time scheduling, and they compete for priority only with other scheduled reports with continuous scheduling.	Yes.
Improve priority scores for reports that have been skipped	The report scheduler lowers priority scores for reports that are skipped, and optimizes for infrequently-scheduled reports, so they are lowered more than frequently-scheduled reports with the same number of skips.	This rule gives infrequently scheduled and slow-completing reports that are at a disadvantage because of the first rule, a chance to run. Each time a scheduled report is forced to skip a scheduled run, its priority score improves slightly. After it is skipped several times, its priority score lets it run ahead of the reports that were forcing it to skip.	Yes.
Apply a schedule window to selected reports	To give the report scheduler more flexibility, you can define a schedule window for scheduled reports that do not require a precise start time. The report scheduler adds the schedule window timespan to the priority score for the report, temporarily increasing that score. The window opens at the report's scheduled run time.	The scheduled report's higher priority score allows reports with lower scores to run ahead of it. As time passes and the schedule window narrows, the report's priority score improves until the report is allowed to run.	No. You manually define schedule windows for scheduled reports. The report scheduler does not apply the "Improve priority scores for reports that have been skipped" rule to reports with schedule windows.

How the report scheduler determines the base priority score for a scheduled report

The report scheduler derives a base priority score for all scheduled reports and then adjusts the score for each scheduled report depending on whether it is continuously scheduled, is skipped, or has a schedule window. The base priority score is the average runtime of the scheduled report. The average runtime is based on the stored history of the previous runtimes for the scheduled report. The average runtime is updated with each new run of the report.

To get the base priority score, Splunk software scales up the average runtime for each scheduled report by a default factor of 100. It does this to differentiate between reports with average runtimes that vary by fractions of a second.

The scheduled report history is stored in the KV Store so it is not lost when Splunk Enterprise is shut down and restarted.

Manually assign schedule priorities or schedule windows to problem reports

If you still have problems with skipped scheduled reports after the application of the default report scheduler rules, you can use the Schedule Priority or Schedule Window settings to help resolve these problems by changing how the scheduler prioritizes specific scheduled reports.

Use Schedule Priority to improve the run priority of specific high-value scheduled reports. Use Schedule Window to let other reports run ahead of low-importance scheduled reports.

For an in-depth explanation of these settings, see Prioritize concurrently scheduled reports in Splunk Web.

See Schedule reports to learn how these settings fit into the report scheduling workflow.

Configure the prioritization of scheduled reports through .conf files

The default settings for the rules that manage scheduled report prioritization should keep report skip incidents down to a low level, especially if you apply schedule windows to reports that seem to be causing other reports to skip on a regular basis. However, if you have Splunk Enterprise and get an unacceptable amount of report skipping, adjust the limits.conf parameters that are related to scheduled search prioritization.

Caution Do not change limits.conf settings unless you know what you are doing.

• search_history_max_runtimes: Sets the number of previous runtimes to use to calculate the average runtime for a scheduled report. When this average runtime is scaled up by the priority_runtime_factor, it becomes the priority for the report. Defaults to 10.
• priority_runtime_factor: Sets the factor by which the average runtime of a report is scaled up to get its priority value. Defaults to 10. See "How the report scheduler determines the base priority score for a scheduled report" in this topic.
• priority_skipped_factor: Applies to the "Improve priority scores for reports" rule. It sets the factor by which a report's priority is lowered when it is skipped. Defaults to 1. Set this higher to cause a report's priority to drop faster when it is skipped, or lower, to slow down the priority score drop after skips.

Disable the report scheduler

Disable the report scheduler only when it is necessary. When the report scheduler is disabled, scheduled reports, alerts, and summary-index-populating searches will not run. Actions related to triggered alerts and scheduled reports will not be performed.

Caution If your Splunk Enterprise implementation includes search head clusters, by issuing splunk disable scheduler you disable the scheduler on the cluster captain, no matter which cluster member you issue the command on. The cluster captain manages search scheduling and dispatch across the entire cluster. When you disable report clustering on a cluster captain, you disable report scheduling across the entire cluster.

See "Search head clustering architecture" in the Distributed Search manual.

You might disable the report scheduler to resolve issues around your Splunk Enterprise deployment, such as a heavy processing load or problems related to the network or hardware. You can do this without restarting splunkd.

1. Access the CLI.

See "About the CLI" in the Admin Manual.

2. Enter the splunk disable scheduler command in the CLI.

To enable the report scheduler after it has been disabled, access the CLI and enter the splunk enable scheduler command.

See the report scheduler status

If you are unsure whether the report scheduler is enabled or disabled, access the CLI and enter the show scheduler-status command. This returns the report scheduler status.