Define per-result alerts
Note: This topic explains how to define per-result alerts, one of three types of Splunk alerts. For an overview of the alert types, and more information about getting started with alert creation, go to About alerts, in this manual.
If you want an alert that is triggered whenever a matching result comes in, select a Schedule of Trigger in real-time whenever a result matches from the Schedule step of the Create alert dialog. Then click Next to go to the Actions step.
This "per-result alert" is the most common alert type. It runs in real-time over an "all-time" timespan. It is designed to always alert whenever the search returns a result.
Keep in mind that "events" are not exactly the same as "results": an event is a type of result. If the underlying search for the alert is designed to return individual events, that's fine--the alert will trigger each time a matching event is returned. But you can also design searches that return other kinds of results.
For example, say you're tracking logins to an employees-only section of the store and you've been having problems with people trying to guess employee logins. You'd like to set up an alert that is triggered whenever it finds a user that has made more than two login attempts. This search would return not events, but usernames of people who fit the search criteria. Each time the search returns a result (a username), the alert is triggered, which sets off an action, such as the sending of an email with the username to a set of recipients.
Enable actions for a per-result alert
On the Actions step for a per-result alert, you can enable one or more alert actions. These actions are set off whenever the alert is triggered.
There are three kinds of alert actions that you can enable through the Create alert dialog. For Enable actions you can select any combination of:
- Send email - Select to have an email sent to a list of recipients that you define. You can opt to have this email contain the results of the triggering search job (the result that triggered the alert, in other words).
- Run a script - Select to run a shell script that can perform some other action, such as the sending of an SNMP trap notification or the calling of an API. You determine which script is run.
- Show triggered alerts in Alert manager - Have triggered alerts display in the Alert Manager with a severity level that you define. The severity level is non-functional and is for informational purposes only. (Note: In Manager > Searches and Reports, to have trigger records for an alert display in the Alert Manager, you enable the Tracking alert action.)
You can enable any combination of these alert actions for an individual alert.
Note: You can also arrange to have a triggered alert post its results to an RSS feed. To enable this option, go to Manager > Searches and Reports and click the name of the saved search that the alert is based upon. Then, in the Alert actions section, click Enable for Add to RSS.
Important: Before enabling actions, read "Set up alert actions," in this manual. This topic discusses the various alert actions at length and provides important information about their setup. It also discusses options that are only available via the Searches and reports page in Manager, such as the ability to send reports with alert emails in PDF format, RSS feed notification, and summary indexing enablement.
Set up throttling for a per-result alert
On the Actions step for a per-result alert, you can define its throttling rules. You use throttling to reduce the frequency at which an alert is triggered. For example, if your alert is being triggered by very similar events approximately 10 times per minute, you can set up throttling rules that cut that frequency down to a much more manageable rate. Throttling rules are especially important for per-result alerts, because they are based on real-time searches and are triggered each time they find a matching result.
Alert throttling rules enable you to throttle results that share the same field value for a given number of seconds, minutes, or hours. For example, say you have a search that returns results with
username=kfrog every 2-3 minutes or so. You don't want to get these alerts every few minutes; you'd rather not see alerts for any one
username value more than once per hour. So here's what you do when you define an alert for this search: You select Throttling, enter
username under Suppress for results with the same field value, and select 60 minutes for the throttling interval.
Now, say that after this alert is saved and enabled, the alerting real-time search matches a result where
username=cmonster. The alert is triggered by this result and an alert email is sent to you. But for the subsequent hour, all following matching results with
username=cmonster are ignored. After 60 minutes are up, the next matching result with
username=cmonster triggers the alert again and another alert email goes out to you--and then it won't be triggered a third time for that particular
username value until another hour has passed. The Throttling setting ensures you're not swamped by alert emails from results with
username=cmonster (or any other
username value, for that matter).
You can use the Throttling setting to suppress on more than one field. For example, you might set up a per-result alert that throttles events that share the same
For example, you could have a real-time search with a 60 second window that alerts every time an event with "disk error" event appears. If ten events with the "disk error" message come in within that window, five disk error alerts will be triggered--ten alerts within one minute. And if the alert is set up so that an email goes out to a set of recipients each time it's triggered (see the topic "Set up alert actions," in this manual), those recipients probably won't see the stack of alert emails as being terribly helpful.
You can set the Throttling controls so that when one alert of this type is triggered, all successive alerts of the same type are suppressed for the next 10 minutes. When those 10 minutes are up, the alert can be triggered again and more emails can be sent out as a result--but once it is triggered, another 10 minutes have to pass before it can be triggered a third time.
In general, when you set up throttling for a real-time search it's best to start with a throttling period that matches the length of the base search's window, and then expand the throttling period from there, if necessary. This prevents you from getting duplicate notifications for a given event.
Note: Throttling settings are not usually required for scheduled searches, because only one alert is sent out per run of a scheduled search. But if the search is scheduled to run on a very frequent basis (every five minutes, for example), you can set the throttling controls to suppress the alert so that a much larger span of time--say, 60 minutes--has to pass before another alert of the same type is sent.
On the Sharing step for a per-result alert, you can determine how the alert is shared if you have a role that gives you Write access to the knowledge objects in your app (such as the Power or Admin roles).
Sharing rules are the same for all alert types: you can opt to keep the search private, or you can share the alert as read-only to all users of the app you're currently using. For the latter choice, "read-only" means that other users of your current app can see and use the alert, but they can't update its definition via Manager > Searches and reports.
You can find additional permission settings in Manager > Searches and reports. For more information about managing permissions for Splunk knowledge objects (such as alert-enabled searches) read "Manage knowledge object permissions" in the Knowledge Manager Manual.
Define scheduled alerts
This documentation applies to the following versions of Splunk® Enterprise: 5.0, 5.0.1, 5.0.2, 5.0.3, 5.0.4, 5.0.5, 5.0.6, 5.0.7, 5.0.8, 5.0.9, 5.0.10, 5.0.11, 5.0.12, 5.0.13, 5.0.14, 5.0.15, 5.0.16, 5.0.17, 5.0.18