Offset scheduled search start times
If your organization has a large number of scheduled searches that run at the same time, such as every minute or every five minutes, the Search Scheduler can cause your network hardware or indexers to become overloaded. Use the allow_skew
setting in savedsearches.conf
to reduce these disruptions.
When a Splunk implementation has a large number of scheduled searches set to run at a particular minute, the scheduler attempts to run them as soon as possible after the zeroth second of that minute. This mass of searches all starting at approximately the same moment can cause the CPUs of the hardware involved to run at 100% capacity very briefly, a few seconds at most. These CPU spikes are not necessarily a problem for hardware running Splunk but they might produce intra-network traffic in amounts that overload network switches.
The allow_skew
setting offsets the actual start time of a scheduled search from its scheduled run time by a random amount of time. For example, if you have a search that is scheduled to run every minute, allow_skew
might offset it so it actually starts 23 seconds after the minute, each time it runs. When the scheduler applies allow_skew
to a large number of searches that are running on similar schedules, the randomness of the offset for each search contributes to an even distribution of actual start times.
In Splunk Cloud Platform, allow_skew is set to 5 minutes by default.
allow_skew methods
The value you provide for allow_skew
constrains the search scheduler by providing the maximum amount of time that the search can have its start time offset. You can set allow_skew
by maximum time duration or by percentage of the time period.
Method | Definition | Examples |
---|---|---|
Maximum offset duration | Provide a maximum duration by which the search can be offset using a <int><time_unit> construction.
|
40s for a search with a 1 minute period
|
3m for a search with a 5 minute period
| ||
Percentage of period | Provide a percentage that specifies the maximum amount of time to offset the search as a percentage of the scheduled search period. | 50% for a search with a 1 minute period limits the offset to 30 seconds at most
|
100% for a search with a 5 minute period lets the offset fall anywhere within 5 minutes
|
You might want to provide maximum offset durations when you are applying allow_skew
to individual searches and provide a period percentage when you are applying allow_skew
to multiple searches with one setting. See Setting allow_skew at the app or global level.
The offset time determined by the search scheduler remains constant on successive runs of the search. For example, if you set allow_skew
for a search that runs every 5 minutes and the scheduler offsets it to run 2 minutes after its scheduled start time, it uses that same offset amount on each successive run. The offset amount remains constant for a search until it is edited, at which point the scheduler recalculates the offset for the search.
Searches offset by allow_skew always run over their scheduled time ranges. For example, if a search that is scheduled to run over the past 5 minutes every 5 minutes is supposed to run at 2:05 p.m. but is offset such that it actually runs at 2:07 p.m., it will still search over 2:00:00 p.m. to 2:04:59 p.m.
How the search schedule affects the potential schedule offset
When you set allow_skew
for a scheduled search, the period of the scheduled search determines how much freedom that the search scheduler has to offset the search from its schedule.
The search scheduler has the most freedom to offset scheduled searches with schedules that fit the following cron expression patterns:
Cron expression pattern | Definition |
---|---|
* * * * *
|
Every minute |
*/M * * * *
|
Every M minutes, where M > 0 |
0 * * * *
|
Every hour |
0 */H * * *
|
Every H hours, where H > 0 |
0 0 * * *
|
Every day at midnight |
If you set allow_skew
for a search with one of these cron expression patterns, the search scheduler can offset it by up to 100% of its period, depending on the constriant you set with the allow_skew
value.
Searches with schedules that do not fit those patterns can be offset by 60 seconds at most, regardless of their periods. Such searches are more likely to need to run on or very close to the schedule that you have defined for them.
See Use cron expressions for scheduling in the Alerting Manual.
Set allow_skew at the app or global level
Because allow_skew
is designed for situations in which large numbers of concurrently scheduled searches are causing conflict, you might want to apply one allow_skew
setting to multiple searches. You can apply the setting at the app level or at the global level.
Application level | Definition | Method |
---|---|---|
App | Applies an allow_skew setting to all scheduled searches belonging to a specific app.
|
Add an allow_skew setting to the [default] stanza of the local/savedsearches.conf file for that app.
|
Global | Applies an allow_skew setting to all scheduled searches in your Splunk deployment.
|
Add an allow_skew setting to the [default] stanza in $ETC/system/local/savedsearches.conf .
|
When you apply allow_skew
to multiple searches, you can override those settings for specific searches by giving them their own allow_skew
settings. If you want a search to opt out of a app- or global-level allow_skew
setting, add allow_skew=0
to the savedsearches.conf
stanza for that search.
Allow_skew and other search scheduler settings
The allow_skew
setting is not directly related to the Schedule Window and Schedule Priority settings, which help with the management of skipped, concurrently scheduled reports. Do not use allow_skew
as a remedy for that use case.
See Prioritize concurrently-scheduled reports in Splunk Web.
Prioritize concurrently scheduled reports in Splunk Web | Generate PDFs of your reports and dashboards |
This documentation applies to the following versions of Splunk® Enterprise: 7.0.0, 7.0.1, 7.0.2, 7.0.3, 7.0.4, 7.0.5, 7.0.6, 7.0.7, 7.0.8, 7.0.9, 7.0.10, 7.0.11, 7.0.13, 7.1.0, 7.1.1, 7.1.2, 7.1.3, 7.1.4, 7.1.5, 7.1.6, 7.1.7, 7.1.8, 7.1.9, 7.1.10, 7.2.0, 7.2.1, 7.2.2, 7.2.3, 7.2.4, 7.2.5, 7.2.6, 7.2.7, 7.2.8, 7.2.9, 7.2.10, 7.3.0, 7.3.1, 7.3.2, 7.3.3, 7.3.4, 7.3.5, 7.3.6, 7.3.7, 7.3.8, 7.3.9, 8.0.0, 8.0.1, 8.0.2, 8.0.3, 8.0.4, 8.0.5, 8.0.6, 8.0.7, 8.0.8, 8.0.9, 8.0.10, 8.1.1, 8.1.2, 8.1.3, 8.1.4, 8.1.5, 8.1.6, 8.1.7, 8.1.8, 8.1.9, 8.1.13, 8.1.14, 8.2.0, 8.2.1, 8.2.2, 8.2.3, 8.2.4, 8.2.5, 8.2.6, 8.2.7, 8.2.8, 8.2.9, 8.2.10, 8.2.11, 8.2.12, 9.0.0, 9.0.1, 9.0.2, 9.0.3, 9.0.4, 9.0.5, 9.0.6, 9.0.7, 9.0.8, 9.0.9, 9.0.10, 9.1.0, 9.1.1, 9.1.2, 9.1.3, 9.1.4, 9.1.5, 9.1.6, 9.2.0, 9.2.1, 9.2.2, 9.2.3, 9.3.0, 9.3.1, 8.1.0, 8.1.10, 8.1.11, 8.1.12
Feedback submitted, thanks!