Splunk® Enterprise

Alerting Manual

Download manual as PDF

Splunk Enterprise version 5.0 reached its End of Life on December 1, 2017. Please see the migration information.
This documentation does not apply to the most recent version of Splunk. Click here for the latest version.
Download topic as PDF

Define rolling-window alerts

Note: This topic explains how to define scheduled alerts, one of three types of alerts that Splunk provides. For an overview of the alert types, and more information about getting started with alert creation, go to "About alerts," in this manual.

The rolling-window alert type enables you to set up alerts that monitors and evaluates events in real time within a rolling window. The moment that alert conditions are met by the events that are returned within this window, the alert is triggered.

The rolling-window alert type is in some ways a hybrid of the other two alert types (per-result alerts and scheduled alerts). Like the per-result alert type, it is based on a real-time search. However, it isn't triggered each time a matching result is returned by the search. Instead, it evaluates all of the events within the rolling window in real time, and is triggered the moment that specific conditions are met by the events passing through that window, just like a scheduled alert is triggered when specific conditions are met by a scheduled run of its search.

To define a real-time rolling-window alert the Create Alert dialog box guides you through three actions: There are three basic steps for the definition of a real-time rolling window alert:

1. Set the length of the real time window (in the Schedule step).

2. Define the alert triggering conditions (in the Schedule step).

3. Enable the alert actions and define action execution and throttling rules (in the Actions step).

4.3 alerting rollwin schedule.png

For example, you could set up an alert that is triggered whenever there are three failed logins for the same username value over the last 10 minutes (using a real-time search with a 10 minute window). You can also arrange to throttle the alert so that it is not triggered for the same username value more than once an hour.

Set the width of the rolling window

When you define a rolling-window alert, the first thing you do is set the width of the real-time window. Real-time search windows can be set to any number of minutes, hours or days.

  • In the 'Schedule step, select Monitor in real-time over a rolling window of... for the Schedule field.
  • Then, in the fields that appear below the Schedule field define the width of the real-time search window by entering a specific number of minutes, hours, or days.

The alert will monitor events as they pass through this window in real-time. For example, you might have an alert that is triggered whenever any particular user fails to login more than 4 times in a 10 minute span of time. After the alert is set up, various login failure events will pass through this window, but the alert is only triggered when 4 login failures for the same user exist within the span of the 10 minute window at the same time.

If a user experiences three login failures in quick succession, then waits 11 minutes, and then has another login failure, the alert won't be triggered, because the first three events will have passed out of the window by the time the fourth one took place.

Set up triggering conditions

Rolling-window alerts are triggered when the results within their rolling window meet specific conditions such as passing a numerical threshold.

These triggering conditions break rolling-window alerts into two subcategories: basic conditional rolling-window alerts and advanced conditional rolling-window alerts. You define these triggering conditions when you set values for the Trigger if field on the Schedule step of the Create Alert dialog.

The definition of these triggering conditions is handled in exactly the same manner for rolling-window alerts as for scheduled alerts, except that in this case the alert is triggered whenever results within the rolling window meet the specified triggering conditions. (For more information, see the section on definition of alert triggering conditions in the topic "Define scheduled alerts," in this manual.)

For example, in the case of a basic conditional alert setup, where the triggering condition involves the search result count being greater than, less than, equal to, or unequal to a specific number, this condition must exist within the rolling real-time window for the alert to be triggered. If the alert is triggered when the number of results becomes greater than 100, then the alert won't be triggered until 101 results exist within the rolling window at the same time.

Advanced conditional alerts also work in much the same way for rolling-window alerts as they do for scheduled alerts. The only difference is that in this case the secondary, conditional search runs in real time as well. It continuously evaluates the results returned in the time range window of the original real time search. The alert is triggered at the moment when a single result is returned by the conditional search.

Note: How do you deal with a situation where an alert would continue to be triggered with each new result received? To take the basic conditional alert example, what if there's a rush of matching results and the "greater than 100" condition is met by all of them? It could potentially lead to a corresponding rush of alert emails--something that wouldn't be appreciated by their recipients. That's why you use throttling to keep alerts from being triggered too frequently. See the section on configuring throttling for rolling-window alerts, below.

Enable actions

On the Actions step for a rolling-window alert, you can enable one or more alert actions. These actions are set off whenever the alert is triggered.

There are three kinds of alert actions that you can enable through the Create alert dialog. For Enable actions you can select any combination of:

  • Send email - Send an email to a list of recipients that you define. You can opt to have this email contain the results of the triggering search job.
  • Run a script - Run a shell script that can perform some other action, such as the sending of an SNMP trap notification or the calling of an API. You determine which script is run.
  • Show triggered alerts in Alert manager - Have triggered alerts display in the Alert Manager with a severity level that you define. The severity level is non-functional and is for informational purposes only. (Note: In Manager > Searches and Reports, to have trigger records for an alert display in the Alert Manager, you enable the Tracking alert action.)

You can enable any combination of these alert actions for an individual alert.

4.3 alerting enable actions.png

Note: You can also arrange to have Splunk post the result of the triggered alert to an RSS feed. To enable this option, go to Manager > Searches and Reports and click the name of the search that the alert is based upon. Then, in the Alert actions section, click Enable for Add to RSS.

Important: Before enabling actions, read "Set up alert actions," in this manual. This topic discusses the various alert actions at length and provides important information about their setup. It also discusses options that are only available via the Searches and reports page in Manager, such as the ability to send reports with alert emails in PDF format, RSS feed notification, and summary indexing enablement.

Determine how often actions are executed when the rolling-window alert is triggered

When you are setting up an alert based on a real-time search with a rolling window, you use the last two settings on the Actions step--Execute actions on and Throttling to determine how often Splunk executes actions after an alert is triggered.

This functionality works for rolling-window alerts in exactly the same way that it does for scheduled alerts, except that in this case you're dealing with alerts that are being triggered in real time.

You can use Execute actions on to say that once the results in the rolling window meet the conditions required to trigger the alert, the alert actions are carried out once for All results triggering the alert or Each result. You might choose the latter if your search is triggered by a small number of results, or if you are using a script to feed information about each individual result into a machine process.

Execute actions on enables you to say that once an alert is triggered, the alert actions are executed for All results returned by the triggering search, or Each result returned by the triggering search. And then you can choose whether or not these actions should be throttled, and if so, how.

If you select All results, you can say that later alert actions should be throttled for a specific number of seconds, minutes, or hours.

If you select Each result, the throttling rules are different, because when the alert is triggered, multiple actions can be executed, one for each result returned by the search. You can throttle action execution for results that share a particular field value.

For example, say you have an rolling-window alert with a 10-minute window that is set to alert whenever any user has more than 10 password failures within that timeframe. The essentially performs a running count of password fail events per user, and then uses a conditional search to look through those events for users with > 10 password failures.

  • On the Actions step it has Send email and Show triggered results in Alert manager selected.
  • It's set to execute actions on Each result. In this case there should be a single result: a username with a corresponding failed password event count.
  • For Throttling it's set to suppress for results that have the same value of username for an hour. This means that even if a user keeps making failed password attempts every few seconds you won't see more alerts triggered for that same person for another hour.

4.3 alerting rollwin eachresultthrottle.png

So you start the alert and eventually user mpoppins makes more than 10 password attempts within the past 10 minutes. This triggers the alert, which sends out an email with his name and the event count to the list of recipients. The alert is also recorded in the Alert manager. Even though mpoppins keeps on making failed password attempts the throttling setting ensures that the alert won't be triggered again by matching events featuring mpoppins for an hour.

For more examples of alerts that use the All results and Each result settings in conjunction with various throttling configurations, see the corresponding discussion for scheduled alerts in the topic "Define scheduled alerts," in this manual.

Share a rolling-window alert with others

On the Sharing step for rolling-window alert, you can determine how the alert is shared if you have a role that gives you Write access to the knowledge objects in your app (such as the Power or Admin roles).

Sharing rules are the same for all alert types: you can opt to keep the search private, or you can share the alert as read-only to all users of the app you're currently using. For the latter choice, "read-only" means that other users of your current app can see and use the alert, but they can't update its definition via Manager > Searches and reports.

4.3 alerting sharing.png

You can find additional permission settings in Manager > Searches and reports. For more information about managing permissions for Splunk knowledge objects (such as alert-enabled searches) read "Manage knowledge object permissions" in the Knowledge Manager Manual.

PREVIOUS
Define scheduled alerts
  NEXT
Use Manager to update and expand alert functionality

This documentation applies to the following versions of Splunk® Enterprise: 5.0, 5.0.1, 5.0.2, 5.0.3, 5.0.4, 5.0.5, 5.0.6, 5.0.7, 5.0.8, 5.0.9, 5.0.10, 5.0.11, 5.0.12, 5.0.13, 5.0.14, 5.0.15, 5.0.16, 5.0.17, 5.0.18


Was this documentation topic helpful?

Enter your email address, and someone from the documentation team will respond to you:

Please provide your comments here. Ask a question or make a suggestion.

You must be logged into splunk.com in order to post comments. Log in now.

Please try to keep this discussion focused on the content covered in this documentation topic. If you have a more general question about Splunk functionality or are experiencing a difficulty with Splunk, consider posting a question to Splunkbase Answers.

0 out of 1000 Characters