Splunk® Enterprise

Reporting Manual

Download manual as PDF

Download topic as PDF

Offset scheduled search start times

If your organization has a large number of scheduled searches that run at the same time, such as every minute or every five minutes, the Search Scheduler can cause your network hardware or indexers to become overloaded. Use the allow_skew setting in savedsearches.conf to reduce these disruptions.

When a Splunk implementation has a large number of scheduled searches set to run at a particular minute, the scheduler attempts to run them as soon as possible after the zeroth second of that minute. This can cause the CPUs of the hardware involved to run at 100% of capacity very briefly, a few seconds at most. This is not necessarily a problem for hardware running Splunk but it might produce intra-network traffic in amounts that overload network switches.

The allow_skew setting offsets the actual start time of a scheduled search from its scheduled run time by a random amount of time. For example, if you have a search that is scheduled to run every minute, allow_skew might offset it so it actually starts 23 seconds after the minute, each time it runs. When the scheduler applies allow_skew to a large number of searches that are running on similar schedules, the randomness of the offset for each search should contribute to an even distribution of actual start times.

If you have Splunk Cloud, you must file a support ticket to apply allow_skew to your scheduled searches.

The allow_skew setting methods

The value you provide for allow_skew constrains the search scheduler by providing the maximum amount of time that the search can have its start time be offset by. You can set allow_skew in two ways:

Method Definition Examples
Maximum offset duration Provide a maximum duration by which the search can be offset, using a <int><time_unit> construction.
  • 40s for a search with a 1 minute period
  • 3m for a search with a 5 minute period
Percentage of period Provide a percentage that specifies the maximum amount of time to offset the search as a percentage of the scheduled search period
  • 50% for a search with a 1 minute period limits the offset to 30 seconds at most.
  • 100% for a search with a 5 minute period lets the offset fall anywhere within 5 minutes.

You might want to provide maximum offset durations when you are applying allow_skew to individual searches, and provide a period percentage when you are applying allow_skew to multiple searches with one setting. See Setting allow_skew at the app or global level.

The offset time determined by the search scheduler remains constant on successive runs of the search. For example, if you set allow_skew for a search that runs every 5 minutes and the scheduler offsets it to run 2 minutes after its scheduled start time, it uses that same offset amount on each successive run. The offset amount remains constant for a search until it is edited, at which point the scheduler will recalculate the offset for the search.

Searches offset by allow_skew always search over the time ranges that they are designed to search over. In other words, if a search that is supposed to run over the past 5 minutes every 5 minutes is supposed to run at 2:05pm but is offset such that it actually runs at 2:07pm, it will still search over 2:00:00pm to 2:04:59pm.

How the search schedule affects the potential schedule offset

When you set allow_skew for a scheduled search, the period of the scheduled search determines how much freedom that the search scheduler has to offset the search from its schedule.

The search scheduler has the most freedom to offset scheduled searches with schedules that fit the following cron expression patterns:

Cron expression pattern Definition
* * * * * Every minute
*/M * * * * Every M minutes, where M > 0
0 * * * * Every hour
0 */H * * * Every H hours, where H > 0
0 0 * * * Every day at midnight

If you set allow_skew for a search with one of these cron expression patterns, the search scheduler can give it a an offset of up to 100% of its period, depending on the constriant you set with the allow_skew value.

Searches with schedules that do not fit those patterns can only be offset by 60 seconds at most, regardless of their periods. The assumption is that these searches are more likely to need to run on or very close to the schedule that you have defined for them.

See Use cron expressions for scheduling, in the Alerting Manual.

Setting allow_skew at the app or global level

Because allow_skew is designed for situations where large numbers of concurrently scheduled searches are causing trouble, you might want to apply one allow_skew setting to multiple searches. If you do this, you can apply the setting at the app level, or at the global level.

Application level Definition Method
App Applies an allow_skew setting to all scheduled searches belonging to a specific app. Add an allow_skew setting to the [default] stanza of the local/savedsearches.conf file for that app.
Global Applies an allow_skew setting globally, to all scheduled searches in your Splunk deployment. Add an allow_skew setting to the [default] stanza in $ETC/system/local/savedsearches.conf.

When you apply allow_skew to multiple searches in this manner, you can still override those settings for specific searches by giving them their own allow_skew settings. If you want a search to opt out of a app- or global-level allow_skew setting, add allow_skew=0 to the savedsearches.conf stanza for that search.

Allow_skew and other search scheduler settings

The allow_skew setting is not directly related to the Schedule Window and Schedule Priority settings, which help with the management of skipped, concurrently scheduled reports. allow_skew should not be considered as a direct remedy for that use case.

See Prioritize concurrently-scheduled reports in Splunk Web.

PREVIOUS
Prioritize concurrently scheduled reports in Splunk Web
  NEXT
Generate PDFs of your reports and dashboards

This documentation applies to the following versions of Splunk® Enterprise: 6.6.0, 6.6.1, 6.6.2, 6.6.3


Was this documentation topic helpful?

Enter your email address, and someone from the documentation team will respond to you:

Please provide your comments here. Ask a question or make a suggestion.

You must be logged into splunk.com in order to post comments. Log in now.

Please try to keep this discussion focused on the content covered in this documentation topic. If you have a more general question about Splunk functionality or are experiencing a difficulty with Splunk, consider posting a question to Splunkbase Answers.

0 out of 1000 Characters