Offset scheduled search start times

If your organization has a large number of scheduled searches that run at the same time, such as every minute or every five minutes, the Search Scheduler can cause your network hardware or indexers to become overloaded. Use the allow_skew setting in savedsearches.conf to reduce these disruptions.

When a Splunk implementation has a large number of scheduled searches set to run at a particular minute, the scheduler attempts to run them as soon as possible after the zeroth second of that minute. This mass of searches all starting at approximately the same moment can cause the CPUs of the hardware involved to run at 100% capacity very briefly, a few seconds at most. These CPU spikes are not necessarily a problem for hardware running Splunk but they might produce intra-network traffic in amounts that overload network switches.

The allow_skew setting offsets the actual start time of a scheduled search from its scheduled run time by a random amount of time. For example, if you have a search that is scheduled to run every minute, allow_skew might offset it so it actually starts 23 seconds after the minute, each time it runs. When the scheduler applies allow_skew to a large number of searches that are running on similar schedules, the randomness of the offset for each search contributes to an even distribution of actual start times.

In Splunk Cloud Platform, allow_skew is set to 5 minutes by default.

allow_skew methods

The value you provide for allow_skew constrains the search scheduler by providing the maximum amount of time that the search can have its start time offset. You can set allow_skew by maximum time duration or by percentage of the time period.

Method	Definition	Examples
Maximum offset duration	Provide a maximum duration by which the search can be offset using a <int><time_unit> construction.	40s for a search with a 1 minute period
		3m for a search with a 5 minute period
Percentage of period	Provide a percentage that specifies the maximum amount of time to offset the search as a percentage of the scheduled search period.	50% for a search with a 1 minute period limits the offset to 30 seconds at most
		100% for a search with a 5 minute period lets the offset fall anywhere within 5 minutes

You might want to provide maximum offset durations when you are applying allow_skew to individual searches and provide a period percentage when you are applying allow_skew to multiple searches with one setting. See Setting allow_skew at the app or global level.

The offset time determined by the search scheduler remains constant on successive runs of the search. For example, if you set allow_skew for a search that runs every 5 minutes and the scheduler offsets it to run 2 minutes after its scheduled start time, it uses that same offset amount on each successive run. The offset amount remains constant for a search until it is edited, at which point the scheduler recalculates the offset for the search.

Searches offset by allow_skew always run over their scheduled time ranges. For example, if a search that is scheduled to run over the past 5 minutes every 5 minutes is supposed to run at 2:05 p.m. but is offset such that it actually runs at 2:07 p.m., it will still search over 2:00:00 p.m. to 2:04:59 p.m.

How the search schedule affects the potential schedule offset

When you set allow_skew for a scheduled search, the period of the scheduled search determines how much freedom that the search scheduler has to offset the search from its schedule.

The search scheduler has the most freedom to offset scheduled searches with schedules that fit the following cron expression patterns:

Cron expression pattern	Definition
* * * * *	Every minute
*/M * * * *	Every M minutes, where M > 0
0 * * * *	Every hour
0 */H * * *	Every H hours, where H > 0
0 0 * * *	Every day at midnight

If you set allow_skew for a search with one of these cron expression patterns, the search scheduler can offset it by up to 100% of its period, depending on the constriant you set with the allow_skew value.

Searches with schedules that do not fit those patterns can be offset by 60 seconds at most, regardless of their periods. Such searches are more likely to need to run on or very close to the schedule that you have defined for them.

See Use cron expressions for scheduling in the Alerting Manual.

Set allow_skew at the app or global level

Because allow_skew is designed for situations in which large numbers of concurrently scheduled searches are causing conflict, you might want to apply one allow_skew setting to multiple searches. You can apply the setting at the app level or at the global level.

Application level	Definition	Method
App	Applies an allow_skew setting to all scheduled searches belonging to a specific app.	Add an allow_skew setting to the [default] stanza of the local/savedsearches.conf file for that app.
Global	Applies an allow_skew setting to all scheduled searches in your Splunk deployment.	Add an allow_skew setting to the [default] stanza in $ETC/system/local/savedsearches.conf.

When you apply allow_skew to multiple searches, you can override those settings for specific searches by giving them their own allow_skew settings. If you want a search to opt out of a app- or global-level allow_skew setting, add allow_skew=0 to the savedsearches.conf stanza for that search.

Allow_skew and other search scheduler settings

The allow_skew setting is not directly related to the Schedule Window and Schedule Priority settings, which help with the management of skipped, concurrently scheduled reports. Do not use allow_skew as a remedy for that use case.

See Prioritize concurrently-scheduled reports in Splunk Web.