Offset scheduled search start times
If your organization has a large number of scheduled searches that run at the same time, such as every minute or every five minutes, the Search Scheduler can cause your network hardware or indexers to become overloaded. Use the
allow_skew setting in
savedsearches.conf to reduce these disruptions.
When a Splunk implementation has a large number of scheduled searches set to run at a particular minute, the scheduler attempts to run them as soon as possible after the zeroth second of that minute. This mass of searches all starting at approximately the same moment can cause the CPUs of the hardware involved to run at 100%j capacity very briefly, a few seconds at most. These CPU spikes are not necessarily a problem for hardware running Splunk but they might produce intra-network traffic in amounts that overload network switches.
allow_skew setting offsets the actual start time of a scheduled search from its scheduled run time by a random amount of time. For example, if you have a search that is scheduled to run every minute,
allow_skew might offset it so it actually starts 23 seconds after the minute, each time it runs. When the scheduler applies
allow_skew to a large number of searches that are running on similar schedules, the randomness of the offset for each search contributes to an even distribution of actual start times.
If you have Splunk Cloud, you must file a support ticket to apply allow_skew to your scheduled searches.
The value you provide for
allow_skew constrains the search scheduler by providing the maximum amount of time that the search can have its start time offset. You can set
allow_skew by maximum time duration or by percentage of the time period.
|Maximum offset duration||Provide a maximum duration by which the search can be offset using a
|Percentage of period||Provide a percentage that specifies the maximum amount of time to offset the search as a percentage of the scheduled search period.|| |
You might want to provide maximum offset durations when you are applying
allow_skew to individual searches and provide a period percentage when you are applying
allow_skew to multiple searches with one setting. See Setting allow_skew at the app or global level.
The offset time determined by the search scheduler remains constant on successive runs of the search. For example, if you set
allow_skew for a search that runs every 5 minutes and the scheduler offsets it to run 2 minutes after its scheduled start time, it uses that same offset amount on each successive run. The offset amount remains constant for a search until it is edited, at which point the scheduler recalculates the offset for the search.
Searches offset by allow_skew always run over their scheduled time ranges. For example, if a search that is scheduled to run over the past 5 minutes every 5 minutes is supposed to run at 2:05 p.m. but is offset such that it actually runs at 2:07 p.m., it will still search over 2:00:00 p.m. to 2:04:59 p.m.
How the search schedule affects the potential schedule offset
When you set
allow_skew for a scheduled search, the period of the scheduled search determines how much freedom that the search scheduler has to offset the search from its schedule.
The search scheduler has the most freedom to offset scheduled searches with schedules that fit the following cron expression patterns:
|Cron expression pattern||Definition|
||Every M minutes, where M > 0|
||Every H hours, where H > 0|
||Every day at midnight|
If you set
allow_skew for a search with one of these cron expression patterns, the search scheduler can offset it by up to 100% of its period, depending on the constriant you set with the
Searches with schedules that do not fit those patterns can be offset by 60 seconds at most, regardless of their periods. Such searches are more likely to need to run on or very close to the schedule that you have defined for them.
See Use cron expressions for scheduling in the Alerting Manual.
Setting allow_skew at the app or global level
allow_skew is designed for situations in which large numbers of concurrently scheduled searches are causing conflict, you might want to apply one
allow_skew setting to multiple searches. You can apply the setting at the app level or at the global level.
|App|| Applies an
|| Add an |
|Global|| Applies an
|| Add an |
When you apply
allow_skew to multiple searches, you can override those settings for specific searches by giving them their own
allow_skew settings. If you want a search to opt out of a app- or global-level
allow_skew setting, add
allow_skew=0 to the
savedsearches.conf stanza for that search.
Allow_skew and other search scheduler settings
allow_skew setting is not directly related to the Schedule Window and Schedule Priority settings, which help with the management of skipped, concurrently scheduled reports. Do not use
allow_skew as a remedy for that use case.
Prioritize concurrently scheduled reports in Splunk Web
Generate PDFs of your reports and dashboards
This documentation applies to the following versions of Splunk® Enterprise: 6.6.0, 6.6.1, 6.6.2, 6.6.3, 6.6.4, 6.6.5, 6.6.6, 6.6.7, 6.6.8, 7.0.0, 7.0.1, 7.0.2, 7.0.3, 7.0.4, 7.1.0, 7.1.1, 7.1.2