Splunk® Enterprise

Alerting Manual

Download manual as PDF

Splunk Enterprise version 5.0 reached its End of Life on December 1, 2017. Please see the migration information.
This documentation does not apply to the most recent version of Splunk. Click here for the latest version.
Download topic as PDF

Define per-result alerts

Note: This topic explains how to define per-result alerts, one of three types of Splunk alerts. For an overview of the alert types, and more information about getting started with alert creation, go to About alerts, in this manual.

If you want an alert that is triggered whenever a matching result comes in, select a Schedule of Trigger in real-time whenever a result matches from the Schedule step of the Create alert dialog. Then click Next to go to the Actions step.

This "per-result alert" is the most common alert type. It runs in real-time over an "all-time" timespan. It is designed to always alert whenever the search returns a result.

Keep in mind that "events" are not exactly the same as "results": an event is a type of result. If the underlying search for the alert is designed to return individual events, that's fine--the alert will trigger each time a matching event is returned. But you can also design searches that return other kinds of results.

For example, say you're tracking logins to an employees-only section of the store and you've been having problems with people trying to guess employee logins. You'd like to set up an alert that is triggered whenever it finds a user that has made more than two login attempts. This search would return not events, but usernames of people who fit the search criteria. Each time the search returns a result (a username), the alert is triggered, which sets off an action, such as the sending of an email with the username to a set of recipients.

Enable actions for a per-result alert

On the Actions step for a per-result alert, you can enable one or more alert actions. These actions are set off whenever the alert is triggered.

There are three kinds of alert actions that you can enable through the Create alert dialog. For Enable actions you can select any combination of:

  • Send email - Select to have an email sent to a list of recipients that you define. You can opt to have this email contain the results of the triggering search job (the result that triggered the alert, in other words).
  • Run a script - Select to run a shell script that can perform some other action, such as the sending of an SNMP trap notification or the calling of an API. You determine which script is run.
  • Show triggered alerts in Alert manager - Have triggered alerts display in the Alert Manager with a severity level that you define. The severity level is non-functional and is for informational purposes only. (Note: In Manager > Searches and Reports, to have trigger records for an alert display in the Alert Manager, you enable the Tracking alert action.)

You can enable any combination of these alert actions for an individual alert.

4.3 alerting enable actions.png

Note: You can also arrange to have a triggered alert post its results to an RSS feed. To enable this option, go to Manager > Searches and Reports and click the name of the saved search that the alert is based upon. Then, in the Alert actions section, click Enable for Add to RSS.

Important: Before enabling actions, read "Set up alert actions," in this manual. This topic discusses the various alert actions at length and provides important information about their setup. It also discusses options that are only available via the Searches and reports page in Manager, such as the ability to send reports with alert emails in PDF format, RSS feed notification, and summary indexing enablement.

Set up throttling for a per-result alert

On the Actions step for a per-result alert, you can define its throttling rules. You use throttling to reduce the frequency at which an alert is triggered. For example, if your alert is being triggered by very similar events approximately 10 times per minute, you can set up throttling rules that cut that frequency down to a much more manageable rate. Throttling rules are especially important for per-result alerts, because they are based on real-time searches and are triggered each time they find a matching result.

Alert throttling rules enable you to throttle results that share the same field value for a given number of seconds, minutes, or hours. For example, say you have a search that returns results with username=cmonster and username=kfrog every 2-3 minutes or so. You don't want to get these alerts every few minutes; you'd rather not see alerts for any one username value more than once per hour. So here's what you do when you define an alert for this search: You select Throttling, enter username under Suppress for results with the same field value, and select 60 minutes for the throttling interval.

4.3 alerting throttling per-result.png

Now, say that after this alert is saved and enabled, the alerting real-time search matches a result where username=cmonster. The alert is triggered by this result and an alert email is sent to you. But for the subsequent hour, all following matching results with username=cmonster are ignored. After 60 minutes are up, the next matching result with username=cmonster triggers the alert again and another alert email goes out to you--and then it won't be triggered a third time for that particular username value until another hour has passed. The Throttling setting ensures you're not swamped by alert emails from results with username=cmonster (or any other username value, for that matter).

You can use the Throttling setting to suppress on more than one field. For example, you might set up a per-result alert that throttles events that share the same clientip and host values.

For example, you could have a real-time search with a 60 second window that alerts every time an event with "disk error" event appears. If ten events with the "disk error" message come in within that window, five disk error alerts will be triggered--ten alerts within one minute. And if the alert is set up so that an email goes out to a set of recipients each time it's triggered (see the topic "Set up alert actions," in this manual), those recipients probably won't see the stack of alert emails as being terribly helpful.

You can set the Throttling controls so that when one alert of this type is triggered, all successive alerts of the same type are suppressed for the next 10 minutes. When those 10 minutes are up, the alert can be triggered again and more emails can be sent out as a result--but once it is triggered, another 10 minutes have to pass before it can be triggered a third time.

In general, when you set up throttling for a real-time search it's best to start with a throttling period that matches the length of the base search's window, and then expand the throttling period from there, if necessary. This prevents you from getting duplicate notifications for a given event.

Note: Throttling settings are not usually required for scheduled searches, because only one alert is sent out per run of a scheduled search. But if the search is scheduled to run on a very frequent basis (every five minutes, for example), you can set the throttling controls to suppress the alert so that a much larger span of time--say, 60 minutes--has to pass before another alert of the same type is sent.

Share per-result alerts with others

On the Sharing step for a per-result alert, you can determine how the alert is shared if you have a role that gives you Write access to the knowledge objects in your app (such as the Power or Admin roles).

Sharing rules are the same for all alert types: you can opt to keep the search private, or you can share the alert as read-only to all users of the app you're currently using. For the latter choice, "read-only" means that other users of your current app can see and use the alert, but they can't update its definition via Manager > Searches and reports.

4.3 alerting sharing.png

You can find additional permission settings in Manager > Searches and reports. For more information about managing permissions for Splunk knowledge objects (such as alert-enabled searches) read "Manage knowledge object permissions" in the Knowledge Manager Manual.

PREVIOUS
About alerts
  NEXT
Define scheduled alerts

This documentation applies to the following versions of Splunk® Enterprise: 5.0, 5.0.1, 5.0.2, 5.0.3, 5.0.4, 5.0.5, 5.0.6, 5.0.7, 5.0.8, 5.0.9, 5.0.10, 5.0.11, 5.0.12, 5.0.13, 5.0.14, 5.0.15, 5.0.16, 5.0.17, 5.0.18


Comments

I agree with Gschmidtz & Araitz.. Splunk is truly far ahead of it's time :)

Sherm77
September 10, 2013

So Splunk contains a time distortion feature. When will you release it :-P

Gschmitz
January 17, 2013

How can kirk/mccoy be getting the same alerts as picard? This represents a rift in the space-time continuum.

Araitz, Splunker
June 18, 2012

Was this documentation topic helpful?

Enter your email address, and someone from the documentation team will respond to you:

Please provide your comments here. Ask a question or make a suggestion.

You must be logged into splunk.com in order to post comments. Log in now.

Please try to keep this discussion focused on the content covered in this documentation topic. If you have a more general question about Splunk functionality or are experiencing a difficulty with Splunk, consider posting a question to Splunkbase Answers.

0 out of 1000 Characters