Prioritize concurrently scheduled reports in Splunk Web

When a scheduled report does not run before its next scheduled start time, the scheduler skips that run of the report and moves on to the next one. Use the Schedule Priority and Schedule Window settings to help ensure that important reports are skipped rarely, if ever.

The Schedule Window and Schedule Priority settings change how the search scheduler prioritizes scheduled reports when your Splunk deployment reaches its system-wide limit for concurrently-scheduled reports.

When the system-wide concurrent report limit is reached, the search scheduler prioritizes the excess concurrently scheduled reports and runs them as the number of concurrently-running scheduled reports dips below the concurrent scheduled search limit. This means that these reports will not run exactly at the time that they are scheduled to run. But when they do run, they still gather data for time range that they would have covered if they started on time. In some cases the search scheduler may need to prioritize certain reports ahead of others to ensure that it obtains current data, or to prevent gaps in data collection.

Use Schedule Priority to improve the run priority of high-value scheduled reports. Use Schedule Window to let other reports run ahead of low-importance scheduled reports.

The Schedule Priority and Schedule Window settings are mutually exclusive. If a scheduled report has both a Schedule Priority of Higher or Highest and a defined Schedule Window, the search scheduler prioritizes the report according to the Schedule Priority value and ignores the Schedule Window definition.

To learn how Schedule Window and Schedule Priority fit into the Splunk Web procedure for scheduled report definition, see Schedule reports.

For more information about limits for concurrently scheduled reports and the search scheduler, see Configure the priority of scheduled reports. This topic discusses additional concurrent report management methods that are only available through configuration file edits.

The Monitoring Console has dashboards that show you how the search scheduler is managing your scheduled searches. If you use Splunk Enterprise, the Scheduler Activity dashboards display your system-wide concurrent search limits for historical and real-time reports. They also display the total number of concurrent searches that you are running at the present moment. See Scheduler Activity in Monitoring Splunk Enterprise.

Search scheduler example

If you have scheduled seven reports to run concurrently and your concurrency limit for scheduled searches is three, then four of those reports will be deferred and prioritized by the search scheduler. The report that is first in the priority will run as soon as one of the concurrently running reports completes, and the other deferred reports will follow as more reports complete their runs.

A scheduled report that has been deferred by the search scheduler will not run on its schedule. When it does run, it will search over the time range it was originally scheduled to cover.

For example, say you have a scheduled report that runs on the hour, over the past hour. Now, say that report gets deferred for its 2 a.m. run by the search scheduler, and instead runs at 2:07 a.m. When the report runs at 2:07 a.m., it still searches over 1 a.m. to 1:59 a.m., the original time range set for that scheduled run of the report.

Use Schedule Priority to raise the priority of scheduled reports

Schedule Priority lets you improve the the run priority of a deferred scheduled report. This means that it may run ahead of other reports, and may have a better chance of running on or near its scheduled run time.

Use the Schedule Priority setting with discretion. It works best if only a small percentage of your reports have Schedule Priority values of Higher or Highest.

Give a report a Schedule Priority of Higher to have it run ahead of other reports of the same scheduling mode. Give a report a scheduling priority of Highest to have it run ahead of reports with lesser priorities, regardless of their scheduling mode.

You can only see the Schedule Priority field and set it to a value other than Default if at least one of your roles has the edit_search_schedule_priority capability. You also need the schedule_search capability to schedule reports.

How the search scheduler prioritizes Schedule Priority and scheduling mode combinations

All manually scheduled reports have a scheduling mode. There are two available scheduling modes: real-time and continuous. Scheduled reports get the real-time scheduling mode by default unless they are enabled for summary indexing, in which case their scheduling mode is changed to continuous.

Scheduled reports with the real-time scheduling mode can potentially skip scheduled runs if enough other scheduled reports take priority over them, while reports with the continuous scheduling mode always run, eventually.

The real-time scheduling mode for scheduled reports is not directly related to real-time searches and has nothing to do with whether a search "runs in real time." Both historical scheduled reports and real-time scheduled reports can use the real-time scheduling mode.

See Real-time scheduling and continuous scheduling.

Here is a more detailed breakdown of how the Schedule Priority settings work with regard to scheduling mode:

• At the Default and Higher priorities, scheduled reports with the real-time scheduling mode are prioritized by the search scheduler ahead of reports with the concurrent scheduling mode. They are prioritized behind scheduled reports with the Highest scheduling mode.
• Among scheduled reports with the continuous scheduling mode, the search scheduler always prioritizes reports with the Higher priority ahead of reports with the Default priority.
• Among scheduled reports with the real-time scheduling mode, the search scheduler always prioritizes reports with the Higher priority ahead of reports with the Default priority.
• The search scheduler always prioritizes scheduled reports with the Highest priority ahead of scheduled reports with the Default and Higher priorities.
• Within the Highest group of scheduled reports, reports with the real-time mode are prioritized ahead of reports with the continuous mode.

This table shows you how scheduled reports with different Schedule Priority and scheduling mode combinations are prioritized by the search scheduler. They are listed in ascending priority order.

Schedule Priority	Scheduling mode	Overall priority order
Default	Continuous	Sixth
Higher	Continuous	Fifth
Default	Real-time	Fourth
Higher	Real-time	Third
Highest	Continuous	Second
Highest	Real-time	First

When you consider how many reports you should have of each Schedule Priority and scheduling mode combination, it may help to think of a pyramid where you have the majority of your scheduled reports at the bottom, and a very small number of scheduled reports at the top. In this pyramid, the lowest-priority combinations, such as Default/continuous and Higher/continuous, would be at the bottom. The highest-priority combinations, such as Highest/continuous and Highest/real-time, would be at the top.

Use Schedule Window to lower the priority of scheduled reports

Schedule Window specifies how long the report scheduler can delay a report from running and allow higher-priority reports to run ahead ahead of it. Set a Schedule Window for a report if:

• It is a low-priority report that does not always have to run at its scheduled run time.
• You have other scheduled reports that run concurrently with the report and are of a higher priority than the report, and you want those other reports to have a better chance of running on or near their scheduled run time.

Reports that are slow to complete and that are run on an infrequent basis are often good candidates for a schedule window.

Capabilities regulate usage of Schedule Window

When you schedule reports, you can see the Schedule Window field only if one of your roles has the edit_search_schedule_window in addition to the schedule_search capability. The Schedule Window field defaults to No Window.

If you have the schedule_search capability but you do not have the edit_search_schedule_window capability, you cannot see the Schedule Window field when you schedule reports. The scheduled reports that you create when you do not have the edit_search_schedule_window capability behave as if they have a Schedule Window set to Auto.

Schedule Window settings

When you set Schedule Window for a scheduled report, you can:

• Select a pre-set time window, such as 5 minutes, 30 minutes, or 4 hours.
• Select Custom and define the width of the schedule window by entering a number of minutes. You can enter any number from 0 to 44,640 (the number of minutes in a 31 day month).
• Select Auto to have the search scheduler calculate the optimal schedule window width for the report, based on the period and average runtime of the report.

The schedule window opens when the report is scheduled to run. Initially the window allows other more important reports to be run ahead of the report. As the schedule window approaches its close, the possibility that the report will run increases.

Never give a report a Schedule Window that is equal to or greater than its period. For example, if you have a report that has a period of 1 hour, give it a Schedule Window value that is under an hour. If you give it a schedule window wider than an hour, the report might miss its scheduled runs.

If you want to verify that a schedule window is being applied to a particular report, search for the report in scheduler.log. If it has a schedule window it will have a nonzero window_time value.