Splunk® Enterprise

Reporting Manual

Download manual as PDF

This documentation does not apply to the most recent version of Splunk. Click here for the latest version.
Download topic as PDF

Configure the priority of scheduled reports

This topic discusses the two options you can use to control the priority of concurrent scheduled reports with the report scheduler (also often referred to as the "search scheduler"). The options are real-time scheduling and continuous scheduling.

  • Real-time scheduling ensures that scheduled reports are always run over the most recent time range, even when a number of reports are scheduled to run at approximately the same time and the scheduler can only run one report at a time. Because of the way it works, reports with real-time scheduling can end up skipping scheduled runs. However, they are always given priority over searches with continuous scheduling.
  • Continuous scheduling ensures that each scheduled run of a report is eventually performed, even if the result is that those reports are delayed. Splunk Enterprise gives all scheduled reports real-time scheduling by default, but when a scheduled report is enabled for summary indexing, Splunk Enterprise automatically changes its scheduling option to continuous.

You can manage settings for both of these scheduler options at the report level in savedsearches.conf.

To understand the necessity of these two scheduler options, you need to understand how the report scheduler handles concurrent reports.

For more information about scheduling reports, see "Schedule reports", in this manual.

This topic also explains how the "auto-summarization" reports that are automatically set up by Splunk Enterprise for report acceleration are handled by by the report scheduler. See the subtopic at the end of this topic for more information.

How the report scheduler handles concurrent reports

The Splunk Enterprise report scheduler limits the number of scheduled reports that can be run concurrently. The default, set by the max_searches_perc setting in limits.conf, sets the maximum number of concurrent searches that can be handled by the scheduler to 50% of the system-wide limits for historical reports.

Splunk Enterprise determines the concurrent historical report limit using this calculation:

System-wide maximum number of concurrent historical reports = (max_searches_per_cpu x number of CPUs) + base_max_searches
max_searches_per_cpu defaults to 1, while base_max_searches defaults to 6. So if you have one CPU, you get 7 historical reports that can be run concurrently.

In the report scheduler, the max_searches_perc setting reduces the system-wide number of historical reports that it can run concurrently by 50%. If your system has only one CPU, the report scheduler can safely run only three scheduled reports at a time (3.5 = 50% of 7).

In addition, ad hoc searches and reports always get priority over scheduled reports. If you kick off a lot of ad hoc searches and reports at the same time that you have reports scheduled to run concurrently, some of those reports might get bumped off schedule to make room for the ad hoc work.

If your scheduler can run only three reports at a time, but you have six reports scheduled to run on an hourly basis over the preceding hour's data, what happens? The scheduler lines up the fourth, fifth, and sixth reports and runs them in consecutive order for the scheduled time period, but each report returns information for the time frame over which it was scheduled to run.

In addition, if you kick off three ad hoc searches at the same time that those six reports are scheduled to run, the report scheduler lines all six of the reports up and runs them in consecutive order as described in the preceding paragraph.

Caution Do not change limits.conf settings unless you know what you are doing.

Example of real-time scheduling versus continuous scheduling

So, given how the scheduler works, how is real-time scheduling different from continuous scheduling, and under what conditions would you prefer one option over the other?

First, say you have two saved, scheduled reports that for the purpose of simplicity we'll call A and B:

  • Report A runs every minute and takes 30 seconds to complete
  • Report B runs every 5 minutes and takes 2 minutes to complete

Let's also say that you have a Splunk Enterprise configuration that enables the report scheduler to run only one report at a time.

Both reports are scheduled to run at 1:05pm.

Time Scheduler action
1:05:00pm The scheduler runs A for the 1:04 to 1:05 period, and schedules it to run again at 1:06pm. It is 1:05:30pm when report A completes.
1:05:30pm The scheduler runs report B. Because it takes 2 minutes to run, report B won't complete until 1:07:30.
1:06:00pm The scheduler wakes up and attempts to run report A, but it cannot run because report B is still in process.
1:06:59pm The scheduler continues to attempt to run report A until 1:06:59. At this point what happens next depends on whether report A is using real-time or continuous scheduling (see below).

If report A is configured to have:

  • real-time scheduling, the scheduler skips the 1:05-1:06 run of the report and schedules the next run of report A for 1:07:00pm (for the 1:06 to 1:07 period). The new report run time is based on the current scheduled run time (1:06:00pm).
  • continuous scheduling, the scheduler does not advance the schedule and attempts to run the report for the 1:05 to 1:06pm period indefinitely, and whatever the eventual report run time is, the next time period that report A would cover would be 1:06 to 1:07pm.

Real-time scheduling is the default for all scheduled reports. It's designed to ensure that the report returns current data. It assumes there won't be any problems if some scheduled reports are skipped, as long as it returns up-to-the minute results in the most recent run of the report.

Continuous scheduling is used for situations where problems arise when there's any gap in the collection of report data. In general this is only important for reports that populate summary indexes, though you may find other uses for it. When a report is enabled for summary indexing, Splunk Enterprise changes its scheduling option to continuous automatically.

Note: For more information about summary indexes and the reports that populate them, see "Use summary indexing for increased reporting efficiency" in the Knowledge Manager Manual.

Configure the realtime_schedule option

The system uses the realtime_schedule option in savedsearches.conf to determine the next run time of a scheduled report. This is set individually for each scheduled report.

realtime_schedule= 0 | 1

  • Set realtime_schedule to 1 to use real-time scheduling. With this setting the scheduler makes sure that it is always running the report over the most recent time range. Because reports can't always run concurrently with others, this means that it may skip some report periods. This is the default value for a scheduled report.
  • Set realtime_schedule to 0 to use continuous scheduling. This setting ensures that scheduled report periods are never skipped. Splunk Enterprise automatically sets this value to 0 for any scheduled report that is enabled for summary indexing.

The scheduler is designed to give reports with real-time scheduling priority over those with continuous scheduling. It always tries to run the real-time scheduled reports first.

Note: You should never set realtime_schedule=1 for a report that populates a summary index, precisely because doing so may cause it to skip report periods. This leads to gaps in the summary index, which makes the summary index unreliable.

Report acceleration and the report scheduler

If you are using report acceleration to speed up your slow-completing reports be aware that this process utilizes scheduled reports as well. It runs them behind the scenes to generate new report acceleration summaries and continues to run them to update those summaries thereafter.

By default, the report scheduler is only allowed to allocate up to 25% of its total bandwidth for report acceleration summary creation and update. We don't recommend that you change this value but if you decide you must, go to limits.conf and change the value of the auto_summary_perc attribute to a number that works better for you.

The report scheduler also runs reports that populate report acceleration summaries at the lowest priority. If these "auto-summarization" reports have a scheduling conflict with user-defined alerts, summary-index reports, and regular scheduled reports, the user-defined alerts and reports always get run first. This means that you may run into situations where a summary isn't being created or updated because Splunk Enterprise is busy running more prioritized reports.

For more information about report acceleration, see "Manage report acceleration," in the Knowledge Manager Manual.

PREVIOUS
Embed scheduled reports
  NEXT
Generate PDFs of your reports and dashboards

This documentation applies to the following versions of Splunk® Enterprise: 6.0, 6.0.1, 6.0.2, 6.0.3, 6.0.4, 6.0.5, 6.0.6, 6.0.7, 6.0.8, 6.0.9, 6.0.10, 6.0.11, 6.0.12, 6.0.13, 6.0.14, 6.0.15, 6.1, 6.1.1, 6.1.2, 6.1.3, 6.1.4, 6.1.5, 6.1.6, 6.1.7, 6.1.8, 6.1.9, 6.1.10, 6.1.11, 6.1.12, 6.1.13, 6.2.0, 6.2.1, 6.2.2, 6.2.3, 6.2.4, 6.2.5, 6.2.6, 6.2.7, 6.2.8, 6.2.9, 6.2.10, 6.2.11, 6.2.12, 6.2.13, 6.2.14, 6.2.15


Comments

Ok, update to my comment -- since by default 50% of total resources is allocated to scheduler, 50% of total scheduler would be 25% of total resources. That is confusing since it is not mentioned in this document.

Tphi
October 29, 2014

The default report scheduler allocation for report acceleration is 50% according to the default/limits.conf file in 6.0.5 -- the documentation indicates 25%<br />From default/limits.conf:<br /># fraction of concurrent scheduler searches to use for auto summarization<br />auto_summary_perc = 50

Tphi
October 29, 2014

Was this documentation topic helpful?

Enter your email address, and someone from the documentation team will respond to you:

Please provide your comments here. Ask a question or make a suggestion.

You must be logged into splunk.com in order to post comments. Log in now.

Please try to keep this discussion focused on the content covered in this documentation topic. If you have a more general question about Splunk functionality or are experiencing a difficulty with Splunk, consider posting a question to Splunkbase Answers.

0 out of 1000 Characters