Splunk® Enterprise

Alerting Manual

Download manual as PDF

This documentation does not apply to the most recent version of Splunk. Click here for the latest version.
Download topic as PDF

Define per-result alerts

Note: This topic explains how to define per-result alerts, one of three types of alerts that Splunk provides. For an overview of the alert types, and more information about getting started with alert creation, read "About alerts" in this manual.

If you want an alert that triggers whenever a matching result arrives, perform the following steps in the Save As Alert window:

1. Click the Real time button next to Alert type.

2. For the trigger condition, select Per-result.

3. Click Next to go to alert actions page.

The "per-result" alert is the most common alert type. It runs in real-time over an "all-time" time span. It is designed to always trigger whenever the search returns a result.

Keep in mind that "events" are not exactly the same as "results": an event is a type of result. If the underlying search for the alert is designed to return individual events, that's fine--the alert triggers each time the search returns a matching event. But you can also design searches that return other kinds of results.

For example, say you're tracking logins to an employees-only section of the store and you've been having problems with people trying to guess employee logins. You'd like to set up an alert that triggers whenever it finds a user that has made more than two login attempts. This search would return not events, but usernames of people who fit the search criteria. Each time the search returns a result (a username), Splunk triggers the alert, which sets off an action, such as the sending of an email with the username to a set of recipients.

Enable actions for a per-result alert

On the alert actions page for a per-result alert, you can enable one or more alert actions. These actions fire whenever the alert triggers.

There are three kinds of alert actions that you can enable through the Save as Alert dialog. For Enable actions you can select any combination of:

  • List in triggered alerts - Have triggered alerts display in the Alert Manager with a severity level that you define. The severity level is non-functional and is for informational purposes only. You can enable any combination of these alert actions for an individual alert.
  • Send email - Select to have Splunk send an email to a list of recipients that you define. You can opt to have this email contain the results of the triggering search job (the result that triggered the alert, in other words).
  • Run a script - Select to have Splunk run a shell script that can perform some other action, such as the sending of an Simple Network Management Protocol (SNMP) trap notification or the calling of an Application Program Interface (API). You can specify the script that runs.


60 saveasalert page2 enableactions listintrigger sendemail.png

Note: You can also arrange to have a triggered alert post its results to an RSS feed. To enable this option, go to Settings > Searches and Reports and click the name of the saved search that the alert is based upon. Then, in the Alert actions section, under, Add to RSS, click Enable.

Important: Before enabling actions, be sure to read "Set up alert actions," in this manual. That topic discusses the various alert actions at length and provides important information about their setup. It also discusses options that are only available via the Searches and reports page in Settings, such as the ability to send reports with alert emails in PDF format, RSS feed notifications, and summary indexing configuration.

Set up throttling for a per-result alert

On the alert actions page for a per-result alert, you can define its throttling rules. You use throttling to reduce the frequency at which an alert triggers. For example, if your alert is being triggered by very similar events approximately 10 times per minute, you can set up throttling rules that cut that frequency down to a much more manageable rate. Throttling rules are especially important for per-result alerts, because they are based on real-time searches and get triggered each time they find a matching result.

Splunk's alert throttling rules enable you to throttle results that share the same field value for a given number of seconds, minutes, or hours. For example, say you have a search that returns results with username=cmonster and username=kfrog every 2-3 minutes or so. You don't want to get these alerts every few minutes; you'd rather not see alerts for any one username value more than once per hour. So here's what you do when you define an alert for this search:

1. Click the checkbox next to Throttling.

2. In the Suppress results with field value field, enter username.

3. In the Suppress actions for listbox, select hour(s).

4. In the adjacent field, type in 2. This sets the throttling interval to 2 hours.

60 saveasalert page2 throttle perresult.png

Now, say that after this alert is saved and enabled, the alerting real-time search matches a result where username=cmonster. This result triggers an alert, and Splunk sends an alert email to you. But for the subsequent hour, Splunk ignores all following matching results with username=cmonster. After 60 minutes pass, the next matching result with username=cmonster triggers the alert again and another alert email goes out to you--and then it won't be triggered a third time for that particular username value until another hour goes by. The Throttling setting ensures you're not swamped by alert emails from results with username=cmonster (or any other username value, for that matter).

You can use the Throttling setting to suppress on more than one field. For example, you might set up a per-result alert that throttles events that share the same clientip and host values.

For example, you could have a real-time search with a 60-second window that alerts every time an event with "disk error" appears. If ten events with the "disk error" message come in within that window, five disk error alerts will be triggered--ten alerts within one minute. And if the alert is set up so that an email goes out to a set of recipients each time it's triggered (see the topic "Set up alert actions," in this manual), those recipients probably won't see the stack of alert emails as being terribly helpful.

You can set the Throttling controls so that when one alert of this type gets triggered, all successive alerts of the same type are suppressed for the next 10 minutes. When those 10 minutes are up, the alert can be triggered again and more emails can be sent out as a result--but once it is triggered, another 10 minutes have to pass before it can be triggered a third time.

In general, when you set up throttling for a real-time search, it's best to start with a throttling period that matches the length of the base search's window, and then expand the throttling period from there, if necessary. This prevents you from getting duplicate notifications for a given event.

The Suppress results with field value field accepts comma-delimited lists of multiple items.

Note: Throttling settings are not usually required for scheduled searches, because only one alert is sent out per run of a scheduled search. But if the search is scheduled to run on a very frequent basis (every five minutes, for example), you can set the throttling controls to suppress the alert so that a much larger span of time--say, 60 minutes--has to pass before Splunk can send out another alert of the same type.

Share per-result alerts with others

You can determine how to share the alert if you have a role that gives you Write access to the knowledge objects in your app (such as the Power or Admin roles).

Sharing rules are the same for all alert types: you can opt to keep the search private, or you can share the alert as read-only to all users of the app you're currently using. For the latter choice, "Read-Only in App" means that other users of your current app can see and use the alert, but they can't update its definition via Settings > Searches and reports.

60 saveasalert page2 sharing.png

You can find additional permission settings in Settings > Searches and reports. For more information about managing permissions for Splunk knowledge objects (such as alert-enabled searches) read "Manage knowledge object permissions" in the Knowledge Manager Manual.

PREVIOUS
About alerts
  NEXT
Define scheduled alerts

This documentation applies to the following versions of Splunk® Enterprise: 6.1, 6.1.1, 6.1.2, 6.1.3, 6.1.4, 6.1.5, 6.1.6, 6.1.7, 6.1.8, 6.1.9, 6.1.10, 6.1.11, 6.1.12, 6.1.13


Comments

The example above is a bit confusing because the example says to enter username as the field to throttle by but then in the screenshot it says userid.

Integratorz
July 3, 2019

Was this documentation topic helpful?

Enter your email address, and someone from the documentation team will respond to you:

Please provide your comments here. Ask a question or make a suggestion.

You must be logged into splunk.com in order to post comments. Log in now.

Please try to keep this discussion focused on the content covered in this documentation topic. If you have a more general question about Splunk functionality or are experiencing a difficulty with Splunk, consider posting a question to Splunkbase Answers.

0 out of 1000 Characters