Offset scheduled search start times
If your organization has a large number of scheduled searches that run at the same time, such as every minute or every five minutes, the Search Scheduler can cause your network hardware or indexers to become overloaded. Use the
allow_skew setting in
savedsearches.conf to reduce these disruptions.
When a Splunk implementation has a large number of scheduled searches set to run at a particular minute, the scheduler attempts to run them as soon as possible after the zeroth second of that minute. This can cause the CPUs of the hardware involved to run at 100% of capacity very briefly, a few seconds at most. This is not necessarily a problem for hardware running Splunk but it might produce intra-network traffic in amounts that overload network switches.
allow_skew setting offsets the actual start time of a scheduled search from its scheduled run time by a random amount of time. For example, if you have a search that is scheduled to run every minute,
allow_skew might offset it so it actually starts 23 seconds after the minute, each time it runs. When the scheduler applies
allow_skew to a large number of searches that are running on similar schedules, the randomness of the offset for each search should contribute to an even distribution of actual start times.
If you have Splunk Cloud, you must file a support ticket to apply allow_skew to your scheduled searches.
The allow_skew setting methods
The value you provide for
allow_skew constrains the search scheduler by providing the maximum amount of time that the search can have its start time be offset by. You can set
allow_skew in two ways:
|Maximum offset duration|| Provide a maximum duration by which the search can be offset, using a
|Percentage of period||Provide a percentage that specifies the maximum amount of time to offset the search as a percentage of the scheduled search period|| |
You might want to provide maximum offset durations when you are applying
allow_skew to individual searches, and provide a period percentage when you are applying
allow_skew to multiple searches with one setting. See Setting allow_skew at the app or global level.
The offset time determined by the search scheduler remains constant on successive runs of the search. For example, if you set
allow_skew for a search that runs every 5 minutes and the scheduler offsets it to run 2 minutes after its scheduled start time, it uses that same offset amount on each successive run. The offset amount remains constant for a search until it is edited, at which point the scheduler will recalculate the offset for the search.
Searches offset by allow_skew always search over the time ranges that they are designed to search over. In other words, if a search that is supposed to run over the past 5 minutes every 5 minutes is supposed to run at 2:05pm but is offset such that it actually runs at 2:07pm, it will still search over 2:00:00pm to 2:04:59pm.
How the search schedule affects the potential schedule offset
When you set
allow_skew for a scheduled search, the period of the scheduled search determines how much freedom that the search scheduler has to offset the search from its schedule.
The search scheduler has the most freedom to offset scheduled searches with schedules that fit the following cron expression patterns:
|Cron expression pattern||Definition|
||Every M minutes, where M > 0|
||Every H hours, where H > 0|
||Every day at midnight|
If you set
allow_skew for a search with one of these cron expression patterns, the search scheduler can give it a an offset of up to 100% of its period, depending on the constriant you set with the
Searches with schedules that do not fit those patterns can only be offset by 60 seconds at most, regardless of their periods. The assumption is that these searches are more likely to need to run on or very close to the schedule that you have defined for them.
See Use cron expressions for scheduling, in the Alerting Manual.
Setting allow_skew at the app or global level
allow_skew is designed for situations where large numbers of concurrently scheduled searches are causing trouble, you might want to apply one
allow_skew setting to multiple searches. If you do this, you can apply the setting at the app level, or at the global level.
|App|| Applies an
|| Add an |
|Global|| Applies an
|| Add an |
When you apply
allow_skew to multiple searches in this manner, you can still override those settings for specific searches by giving them their own
allow_skew settings. If you want a search to opt out of a app- or global-level
allow_skew setting, add
allow_skew=0 to the
savedsearches.conf stanza for that search.
Allow_skew and other search scheduler settings
allow_skew setting is not directly related to the Schedule Window and Schedule Priority settings, which help with the management of skipped, concurrently scheduled reports.
allow_skew should not be considered as a direct remedy for that use case.
Prioritize concurrently scheduled reports in Splunk Web
Generate PDFs of your reports and dashboards
This documentation applies to the following versions of Splunk® Enterprise: 6.6.0, 6.6.1, 6.6.2, 6.6.3