Configure the priority of scheduled reports

The Splunk software's report scheduler manages scheduled reports and alerts. Based on your system configuration, the report scheduler sets a limit on the number of reports that can run concurrently. When more reports are scheduled to run than can run concurrently, it prioritizes the excess reports and tries to run them in priority order.

The report scheduler sets limits to ensure that few reports get skipped disproportionally more than others. Reports can be skipped when slow-to-complete scheduled reports crowd out quick-to-complete reports and cause them to miss their scheduled run times.

Why read this topic?

Read this topic when you:

• Have several reports scheduled to run concurrently, and you discover that some of them are consistently skipping their scheduled runs.
• Want to have a detailed understanding of how the scheduling of concurrent reports is managed.
• Need to disable the report scheduler.

For information about how to manually set up a scheduled report, see "Schedule reports" in this manual.

The priority order of different types of searches

The report scheduler determines the run priority for your scheduled reports and alerts. This is important when your resources are limited and only a few reports can run concurrently. The report scheduler runs the reports that it can run at the current time and arranges for the rest to run in a priority order.

The table shows the high level priority in which different types of searches and reports are run.

Priority	Search or Report Type	Description
First priority	Ad hoc historical searches	Historical searches that you run manually always run first. If you start several ad hoc searches at the same time that a number of reports are scheduled to run, some of those reports might get bumped off their schedules to make room for the ad hoc search jobs. For example, your system can run only three scheduled reports concurrently. It is 12 P.M., and you have three scheduled reports that run hourly. A few seconds before noon, you manually run three historical searches. The report scheduler lets your ad hoc searches run concurrently and then queues up the three scheduled reports to run over their scheduled time periods as the ad hoc searches finish.
Second priority	Manually scheduled reports and alerts with real-time scheduling	Reports that you schedule manually use the real-time scheduling mode by default. See "Real-time scheduling and continuous scheduling," in this topic. Within this search and report category, additional priority rules are applied to reduce instances where runs of manually scheduled reports and alerts are skipped. See "How the report scheduler prevents skipped report runs," in this topic.
Third priority	Manually scheduled reports with continuous scheduling	The continuous scheduling mode is used by scheduled reports that populate summary indexes as well as other scheduled reports that cannot have a gap in the collection of report data. See "Real-time scheduling and continuous scheduling," in this topic.
Last priority	Automatically scheduled reports	The processes behind report acceleration and data model acceleration use scheduled reports. These reports are scheduled when you set up acceleration for a report or data model. They generate and update acceleration summaries. If these "auto-summarization" reports have a scheduling conflict with ad hoc searches, alerts, and manually scheduled reports, the auto-summarization reports always runs last. This means that you might have situations where a summary is not being created or updated because other reports with higher priority are running. See "Limits for concurrent scheduled reports," in this topic.

How the report scheduler determines the concurrent scheduled report limit

If you have Splunk Cloud, your deployment is optimized for the performance of searches and reports, and you can work with Splunk Support as required to ensure you are getting desired results. In Splunk Enterprise, you can optimize performance manually, and to do so requires an understanding of the scheduling logic described in this section.

The Splunk report scheduler limits the number of scheduled reports that can be run concurrently. Without these limits, your report performance suffers. To determine what this limit should be, the report scheduler starts with the system-wide limit for concurrent ad hoc historical searches.

See "Accomodate many simultaneous searches" in the Capacity Planning Manual for information about the impact of concurrent searches on search performance.

Caution: If you have Splunk Enterprise, do not change limits.conf settings unless you know what you are doing.

Splunk software determines its system-wide concurrent historical search limit through a calculation where the primary variable is the number of CPUs in the deployment. The calculation includes two parameters that are defined in "limits.conf" in the Admin Manual:

• max_searches_per_cpu is the maximum number of concurrent historical searches allowed per CPU. Defaults to 1.
• base_max_searches is a baseline constant to add to the maximum number of searches, computed as a multiplier of the CPUs. Defaults to 6.

The calculation is as follows:

System-wide maximum number of concurrent historical searches = (max_searches_per_cpu x number of CPUs) + base_max_searches

If your system has one CPU, you can concurrently run a maximum of 7 historical searches ((1 x 1) + 6 = 7).

The report scheduler feeds the system-wide limit for historical searches into another limits.conf parameter: max_searches_perc. The max_searches_perc parameter sets the top number of scheduled reports that can be run concurrently by the report scheduler to 50% of the system-wide limit for concurrent historical searches.

If your system has one CPU, the report scheduler can safely run only three scheduled reports at a time (50% of 7 = 3.5).

The max_searches_perc parameter can be set up so that it allows more or less concurrent scheduled reports depending on the time of day or week. See "Have maximum concurrent scheduled report limits vary by time" next.

Have maximum concurrent scheduled report limits vary by time

You can configure max_searches_perc.n parameter in limits.conf to have regular periods when more or fewer concurrent scheduled reports are allowed. You can use max_searches_perc.n and max_searches_perc.n.when to set up alternative percentages for specific cron periods.

For example:

# The default limit, used when the periods defined below are not in effect.
max_searches_perc = 50 
 
# Increase the value between midnight-5AM.
max_searches_perc.0 = 75
max_searches_perc.0.when = * 0-5 * * *
 
# More specifically, increase it even more on weekends.
max_searches_perc.1 = 90
max_searches_perc.1.when = * 0-5 * * 0,6

The n notation can be any nonnegative number up to 255. The scheduler looks in reverse n order for the first match. If no n settings are provided or if no when matches the current time, the value falls back to the default value for max_searches_perc.

Limits for concurrent scheduled reports that populate acceleration summaries

The auto_summary_perc setting does for automatically-scheduled reports what the max_searches_perc setting does for manually scheduled reports. By default, it sets the top number of automatically scheduled reports that can be run concurrently by the report scheduler to 50% of the system-wide limit for concurrent historical searches.

Use the auto_summary_perc.n parameter to set time intervals when more or less concurrent automatically scheduled reports are allowed. It follows the same logic as the max_searches_perc.n parameter described in "Have maximum concurrent scheduled report limits vary by time."

See "Manage report acceleration" and "Accelerate data models" in the Knowledge Manager Manual.

Real-time scheduling and continuous scheduling

Each manually scheduled report has a scheduling mode. There are two possible scheduling modes: real-time scheduling or continuous scheduling.

Real-time scheduling ensures that the most recent run of the report returns current data. A scheduled report with real-time scheduling runs at its scheduled run time, or not at all. Real-time scheduled reports can be skipped because longer-running reports have not finished, or because multiple reports with real-time scheduling are set to run at the same time, and their number is higher than the concurrent scheduled report limit.

The report scheduler always prioritizes reports with real-time scheduling over reports with continuous scheduling.

Use continuous scheduling for situations where there cannot be a gap in the collection of report data. Continuous scheduling ensures that each scheduled run of a report is eventually performed. If a report with continuous scheduling cannot run now, it will run in the future, after other reports finish.

The report scheduler gives all scheduled reports the real-time scheduling mode by default, unless they are enabled for summary indexing, in which case it changes their scheduling mode to continuous. It does this because summary indexes are not trustworthy when the scheduled reports that populate them are skipped. See "Use summary indexing for increased reporting efficiency" in the Knowledge Manager Manual.

Splunk Enterprise outages and continuously scheduled reports

When you shut down Splunk Enterprise, scheduled reports can miss scheduled runtimes. This is typically not a problem for reports with real-time scheduling, but it is big issue for reports with continuous scheduling, because they cannot miss any scheduled runs.

The report scheduler can replace the missed runs of a continuously scheduled report after Splunk Enterprise goes back online. As long as the report was run at least once on its schedule before the Splunk Enterprise instance went down, the report scheduler can replace its missed report runs. The report scheduler uses the record of the report's last runtime to determine which runs were missed. By default it looks back no more than 24 hours for missed reports. This limit is controlled by the max_continuous_scheduled_search_lookback parameter in limits.conf.

If Splunk Enterprise is shut down before the first scheduled run of a continuously scheduled report, when service is restored the report scheduler cannot make up for the missed runs because it does not have a last-runtime record for the report.

Scheduling mode example

How is the real-time scheduling mode different from the continuous scheduling mode, and under what conditions would you prefer one option over the other?

In this example, you have two scheduled reports that are called A and B:

• Report A runs every minute and takes 30 seconds to finish.
• Report B runs every 5 minutes and takes 2 minutes to finish.

Your Splunk Enterprise configuration enables the report scheduler to run only one report at a time.

Both reports are scheduled to run at 1:05 P.M.

Time	Scheduler action
1:05:00 P.M.	The scheduler runs report A for the 1:04 to 1:05 period and schedules it to run again at 1:06 P.M. It is 1:05:30 P.M. when report A finishes.
1:05:30 P.M.	The scheduler runs report B. Because it takes 2 minutes to run, report B will not finish until 1:07:30.
1:06:00 P.M.	The scheduler wakes up and attempts to run report A, but it cannot run because report B is still in process.
1:06:59 P.M.	The scheduler continues to attempt to run report A until 1:06:59. At this point what happens next depends on whether report A is using real-time or continuous scheduling (see below).

If report A is configured to have:

• Real-time scheduling, the scheduler skips the 1:05-1:06 run of the report and schedules the next run of report A for 1:07:00 P.M. (for the 1:06 to 1:07 period). The new report run time is based on the current scheduled run time (1:06:00 P.M.).
• Continuous scheduling, the scheduler does not advance the schedule and attempts to run the report for the 1:05 to 1:06 P.M. period indefinitely. Whatever the eventual report run time is, the next time period that report A covers is 1:06 to 1:07 P.M.

Configure the scheduling mode for individual reports

The report scheduling mode is defined at the individual report level in savedsearches.conf. See the Admin Manual. Use the realtime_schedule parameter to manually change the scheduling mode for a scheduled report.

realtime_schedule= 0 | 1

• Set realtime_schedule to 1 to use the real-time scheduling mode. This mode forces the scheduled report to run at its scheduled start times. If it cannot run, it skips that scheduled run. This is the default scheduling mode for new scheduled reports.
• Set realtime_schedule to 0 to use the continuous scheduling mode. This mode ensures that the scheduled report never skips a scheduled run. If it cannot run on time, then it will run later. When you enable summary indexing for a scheduled report, Splunk software sets its realtime_schedule value to 0.

The report scheduler gives reports with real-time scheduling priority over those with continuous scheduling.

How the scheduler prevents skipped report runs

The real-time scheduling mode can cause reports to be skipped. To mitigate this, the report scheduler applies a set of rules over the scheduling modes. These rules reduce the number of skipped report runs.

The report scheduler applies three of these rules to all scheduled reports by default. You can manually apply the fourth rule, a scheduling window, to selected reports that do not need to run at their scheduled start times. This gives the report scheduler more flexibility to schedule reports that require more precision.

Rule	Description	Result	Default for all scheduled reports?
Apply a scheduled report priority score	The report scheduler gives each scheduled report a priority score based on its average runtime. It prioritizes lower scores (lower average runtimes) over higher scores (higher average runtimes).	When two or more reports are scheduled to run at the same time, the report scheduler runs the report with the fastest average completion time first. Quick-completing reports are not skipped as often, but slow-completing reports get skipped more.	Yes.
Prioritize real-time scheduling over continuous scheduling	The report scheduler makes sure that scheduled reports with real-time scheduling have better priority scores than scheduled reports that use continuous scheduling.	Reports with real-time scheduling compete for priority only with other scheduled reports with real-time scheduling. Reports with continuous scheduling always get run after reports with real-time scheduling, and they compete for priority only with other scheduled reports with continuous scheduling.	Yes.
Improve priority scores for reports that have been skipped	The report scheduler lowers priority scores for reports that are skipped, and optimizes for infrequently-scheduled reports, so they are lowered more than frequently-scheduled reports with the same number of skips.	This rule gives infrequently scheduled and slow-completing reports that are at a disadvantage because of the first rule, a chance to run. Each time a scheduled report is forced to skip a scheduled run, its priority score improves slightly. After it is skipped several times, its priority score lets it run ahead of the reports that were forcing it to skip.	Yes.
Apply a schedule window to selected reports	To give the report scheduler more flexibility, you can define a schedule window for scheduled reports that do not require a precise start time. The report scheduler adds the schedule window timespan to the priority score for the report, temporarily increasing that score. The window opens at the report's scheduled run time.	The scheduled report's higher priority score allows reports with lower scores to run ahead of it. As time passes and the schedule window narrows, the report's priority score improves until the report is allowed to run.	No. You manually define schedule windows for scheduled reports. The report scheduler does not apply the "Improve priority scores for reports that have been skipped" rule to reports with schedule windows.

How the report scheduler determines the base priority score for a scheduled report

The report scheduler derives a base priority score for all scheduled reports and then adjusts the score for each scheduled report depending on whether it is continuously scheduled, is skipped, or has a schedule window. The base priority score is the average runtime of the scheduled report. The average runtime is based on the stored history of the previous runtimes for the scheduled report. The average runtime is updated with each new run of the report.

To get the base priority score, Splunk software scales up the average runtime for each scheduled report by a default factor of 100. It does this to differentiate between reports with average runtimes that vary by fractions of a second.

The scheduled report history is stored in the KV Store so it is not lost when Splunk Enterprise is shut down and restarted.

Manually assign schedule priorities or schedule windows to problem reports

If you still have problems with skipped scheduled reports after the application of the default report scheduler rules, you can use the Schedule Priority or Schedule Window settings to help resolve these problems by changing how the scheduler prioritizes specific scheduled reports.

Use Schedule Priority to improve the run priority of specific high-value scheduled reports. Use Schedule Window to let other reports run ahead of low-importance scheduled reports.

For an in-depth explanation of these settings, see Prioritize concurrently scheduled reports in Splunk Web.

See Schedule reports to learn how these settings fit into the report scheduling workflow.

Configure the prioritization of scheduled reports through .conf files

The default settings for the rules that manage scheduled report prioritization should keep report skip incidents down to a low level, especially if you apply schedule windows to reports that seem to be causing other reports to skip on a regular basis. However, if you have Splunk Enterprise and get an unacceptable amount of report skipping, adjust the limits.conf parameters that are related to scheduled search prioritization.

Caution Do not change limits.conf settings unless you know what you are doing.

• search_history_max_runtimes: Sets the number of previous runtimes to use to calculate the average runtime for a scheduled report. When this average runtime is scaled up by the priority_runtime_factor, it becomes the priority for the report. Defaults to 10.
• priority_runtime_factor: Sets the factor by which the average runtime of a report is scaled up to get its priority value. Defaults to 10. See "How the report scheduler determines the base priority score for a scheduled report" in this topic.
• priority_skipped_factor: Applies to the "Improve priority scores for reports" rule. It sets the factor by which a report's priority is lowered when it is skipped. Defaults to 1. Set this higher to cause a report's priority to drop faster when it is skipped, or lower, to slow down the priority score drop after skips.

Disable the report scheduler

Disable the report scheduler only when it is necessary. When the report scheduler is disabled, scheduled reports, alerts, and summary-index-populating searches will not run. Actions related to triggered alerts and scheduled reports will not be performed.

Caution If your Splunk Enterprise implementation includes search head clusters, by issuing splunk disable scheduler you disable the scheduler on the cluster captain, no matter which cluster member you issue the command on. The cluster captain manages search scheduling and dispatch across the entire cluster. When you disable report clustering on a cluster captain, you disable report scheduling across the entire cluster.

See "Search head clustering architecture" in the Distributed Search manual.

You might disable the report scheduler to resolve issues around your Splunk Enterprise deployment, such as a heavy processing load or problems related to the network or hardware. You can do this without restarting splunkd.

1. Access the CLI.

See "About the CLI" in the Admin Manual.

2. Enter the splunk disable scheduler command in the CLI.

To enable the report scheduler after it has been disabled, access the CLI and enter the splunk enable scheduler command.

See the report scheduler status

If you are unsure whether the report scheduler is enabled or disabled, access the CLI and enter the show scheduler-status command. This returns the report scheduler status.