Splunk® Enterprise

Alerting Manual

Download manual as PDF

This documentation does not apply to the most recent version of Splunk. Click here for the latest version.
Download topic as PDF

Define rolling-window alerts

Note: This topic explains how to define scheduled alerts, one of three types of alerts that Splunk provides. For an overview of the alert types, and more information about getting started with alert creation, go to "About alerts," in this manual.

The rolling-window alert type enables you to set up alerts that monitors and evaluates events in real time within a rolling window. The moment that alert conditions are met by the events that are returned within this window, the alert triggers.

The rolling-window alert type is in some ways a hybrid of the other two alert types (per-result alerts and scheduled alerts). Like the per-result alert type, it is based on a real-time search. However, it doesn't trigger each time the search returns a matching result. Instead, it evaluates all of the events within the rolling window in real time, and triggers the moment that specific conditions are met by the events passing through that window, just like a scheduled alert triggers when specific conditions are met by a scheduled run of its search.

To define a real-time rolling-window alert the Save As Alert dialog box guides you through three actions: There are three basic steps for the definition of a real-time rolling window alert:

1. Ensure that the Real time button is selected.

2. In the Trigger condition dropdown, select either Number of Results or Custom condition.

3. For either option, set the width of the rolling window by entering a number in the in field and selecting a period of minutes, hours, or days in the adjacent drop-down.

4. Enable the alert actions and define action execution and throttling rules (in the alert actions page).

60 saveasalert realtime rollingwindow.png

For example, you could set up an alert that triggers whenever there are three failed logins for the same username value over the last 10 minutes (using a real-time search with a 10 minute window). You can also arrange to throttle the alert so that it does not trigger for the same username value more than once an hour.

Set the width of the rolling window

When you define a rolling-window alert, you must set the width of the real-time window. Real-time search windows can be set to any number of minutes, hours or days.

  • In the 'Save As Alert page, select Real time for the Alert Type control.
  • Then, in the in control that appears below the Alert Type control, define the width of the real-time search window by entering a specific number of minutes, hours, or days.

The alert will monitor events as they pass through this window in real-time. For example, you might have an alert that triggers whenever any particular user fails to log in more than 4 times in a 10 minute period. After you set up the alert, various login failure events pass through this window, but the alert only triggers when 4 login failures for the same user exist within the span of the 10 minute window at the same time.

If a user experiences three login failures in quick succession, then waits 11 minutes, and then has another login failure, the alert won't trigger, because the first three events will have passed out of the window by the time the fourth one took place.

Set up triggering conditions

Rolling-window alerts trigger when the results within their rolling window meet specific conditions such as passing a numerical threshold.

These triggering conditions break rolling-window alerts into two subcategories: basic conditional rolling-window alerts and advanced conditional rolling-window alerts. You define these triggering conditions when you set values for the Trigger condition field in the Save As Alert dialog.

The definition of these triggering conditions is handled in exactly the same manner for rolling-window alerts as it is for scheduled alerts. The only exception is that, in this case, the alert triggers whenever results within the rolling window meet the specified triggering conditions. (For more information, see the section on definition of alert triggering conditions in the topic "Define scheduled alerts," in this manual.)

For example, in the case of a basic conditional alert setup, where the triggering condition involves the search result count being greater than, less than, equal to, or unequal to a specific number, this condition must exist within the rolling real-time window for the alert to be triggered. If the alert is configured to trigger when the number of results becomes greater than 100, then it won't trigger until 101 results exist within the rolling window at the same time.

Advanced conditional alerts also work in much the same way for rolling-window alerts as they do for scheduled alerts. The only difference is that in this case the secondary, conditional search runs in real time as well. It continuously evaluates the results returned in the time range window of the original real time search. The alert triggers at the moment when a conditional search returns a single result.

Note: How do you deal with a situation where an alert would continue to be triggered with each new result received? To take the basic conditional alert example, what if there's a rush of matching results and the "greater than 100" condition is met by all of them? It could potentially lead to a corresponding rush of alert emails--something that wouldn't be appreciated by their recipients. That's why you use throttling to keep alerts from triggering too frequently. See the section on configuring throttling for rolling-window alerts, below.

Enable actions

On the alert actions page for a rolling-window alert, you can enable one or more alert actions. These actions fire whenever the alert triggers.

There are three kinds of alert actions that you can enable through the Save As Alert dialog. For Enable actions you can select any combination of:

  • List in Triggered Alerts - Have triggered alerts display in the Alert Manager with a severity level that you define. The severity level is non-functional and is for informational purposes only. (Note: In Settings > Searches and Reports, to have trigger records for an alert display in the Alert Manager, you enable the List in Triggered Alerts alert action.)

You can enable any combination of these alert actions for an individual alert.

  • Send email - Send an email to a list of recipients that you define. You can opt to have this email contain the results of the triggering search job.
  • Run a script - Run a shell script that can perform some other action, such as the sending of a Simple Network Management Protocol (SNMP) trap notification or the calling of an API. You determine which script runs.

60 saveasalert page2 enableactions listintrigger sendemail.png

Note: You can also arrange to have Splunk post the result of the triggered alert to an RSS feed. To enable this option, go to Settings > Searches and Reports and click the name of the search that the alert is based upon. Then, in the Alert actions section, under Add to RSS, click Enable.

Important: Before enabling actions, read "Set up alert actions," in this manual. This topic discusses the various alert actions at length and provides important information about their setup. It also discusses options that are only available via the Searches and reports page in Manager, such as the ability to send reports with alert emails in PDF format, RSS feed notification, and summary indexing enablement.

Determine how often actions execute when the rolling-window alert triggers

When you are setting up an alert based on a real-time search with a rolling window, you use the last two settings on the alert actions page--When triggered, execute actions and Throttle? to determine how often Splunk executes actions after an alert triggers.

This functionality works for rolling-window alerts in exactly the same way that it does for scheduled alerts, except that in this case you're dealing with alerts that are being triggered in real time.

You can use When triggered, execute actions to say that once the results in the rolling window meet the conditions required to trigger the alert, the alert actions are carried out Once or For each result. You might choose the latter if your search is triggered by a small number of results, or if you are using a script to feed information about each individual result into a machine process.

When triggered, execute actions enables you to say that, once an alert triggers, the alert actions execute Once for all results returned by the triggering search, or For each result returned by the triggering search. And then you can choose whether or not these actions should be throttled, and if so, how.

If you select Once, you can say that later alert actions should be throttled for a specific number of seconds, minutes, or hours.

If you select For each result, the throttling rules are different, because when the alert triggers, multiple actions can be executed, one for each result returned by the search. You can throttle action execution for results that share a particular field value.

For example, say you have an rolling-window alert with a 10-minute window that is set to alert whenever any user has more than 10 password failures within that timeframe. The essentially performs a running count of password fail events per user, and then uses a conditional search to look through those events for users with greater than 10 password failures.

  • On the alert actions page, the alert has List in Triggered Alerts and Send email selected.
  • It's set to execute actions 'For each result. In this case, there should be a single result: a username with a corresponding failed password event count.
  • For Throttle? it's configured to suppress for results that have the same value of username for an hour. This means that even if a user keeps making failed password attempts every few seconds you won't see more alerts triggered for that same person for another hour.

60 saveasalert page2 throttle perresult.png

So you start the alert and eventually user mpoppins makes more than 10 password attempts within the past 10 minutes. This triggers the alert, which sends out an email with his name and the event count to the list of recipients. The Alert manager also records the alert. Even though mpoppins keeps on making failed password attempts the throttling setting ensures that the alert won't be triggered again by matching events featuring mpoppins for an hour.

For more examples of alerts that use the Once and For each result settings in conjunction with various throttling configurations, see the corresponding discussion for scheduled alerts in the topic "Define scheduled alerts," in this manual.

PREVIOUS
Define scheduled alerts
  NEXT
Update and expand alert functionality in Settings

This documentation applies to the following versions of Splunk® Enterprise: 6.1, 6.1.1, 6.1.2, 6.1.3, 6.1.4, 6.1.5, 6.1.6, 6.1.7, 6.1.8, 6.1.9, 6.1.10, 6.1.11, 6.1.12, 6.1.13


Was this documentation topic helpful?

Enter your email address, and someone from the documentation team will respond to you:

Please provide your comments here. Ask a question or make a suggestion.

You must be logged into splunk.com in order to post comments. Log in now.

Please try to keep this discussion focused on the content covered in this documentation topic. If you have a more general question about Splunk functionality or are experiencing a difficulty with Splunk, consider posting a question to Splunkbase Answers.

0 out of 1000 Characters