Splunk® Enterprise

Alerting Manual

Download manual as PDF

Splunk Enterprise version 5.0 reached its End of Life on December 1, 2017. Please see the migration information.
This documentation does not apply to the most recent version of Splunk. Click here for the latest version.
Download topic as PDF

About alerts

If you've been using Splunk's Search app for a while, you know how you can use its powerful search capabilities to learn all kinds of things about the machine data in your system. But this doesn't help you with the myriad of recurring situations that everyone in IT is faced with on a regular basis. You can't be running searches yourself to find these events all of the time.

This is why we've designed Splunk Enterprise to be the most flexible monitoring tool in your arsenal. You can configure a variety of alerting scenarios for your real-time and historical searches. You can have your historical searches run automatically on regular schedules, and you can set up both types of searches so they send alert messages to you and others when their results meet specific circumstances. You can base these alerts on a wide range of threshold and trend-based scenarios, including empty shopping carts, brute force firewall attacks, and server system errors.

In this manual you'll find:

The three alert categories

Splunk alerts are based on saved searches that run on a regular interval over a set historical time range or in real time (if the saved search is a real-time search). When they are triggered, different actions can take place, such as the sending of an email with the results of the triggering search to a predefined list of people.

There are three broad categories of alerts:

Type of alert Base search is a... Description Alert examples
Alerts based on real-time searches that are triggered every time the base search returns a result. Real-time search (runs over all time) Use this alert type if you need to know the moment a matching result comes in. Useful if you need to design an alert for machine consumption (such as a workflow-oriented application). You can also throttle these alerts to ensure that they aren't triggered too frequently. Referred to as a "per-result alert."
  • Trigger an alert for every failed login attempt, but alert at most once an hour for any given username.
  • Trigger an alert when a "file system full" error occurs on any host, but only send notifications for any given host once per 30 minutes.
Alerts based on historical searches that run on a regular schedule. Historical search This alert type triggers whenever a scheduled run of a historical search returns results that meet a particular condition that you have configured in the alert definition. Best for cases where immediate reaction to an alert is not a priority. You can use throttling to reduce the frequency of redundant alerts. Referred to as a "scheduled alert."
  • Trigger an alert whenever the number of items sold in the previous day is less than 500.
  • Trigger an alert when the number of 404 errors in any 1 hour interval exceeds 100.
Alerts based on real-time searches that monitor events within a rolling time "window". Real-time search Use this alert type to monitor events in real time within a rolling time window of a width that you define, such as a minute, 10 minutes, or an hour. The alert is triggered when its conditions are met by events as they pass through this window in real time. You can throttle these alerts to ensure that they aren't triggered too frequently. Referred to as a "rolling-window alert."
  • Trigger an alert whenever there are three consecutive failed logins for a user between now and 10 minutes ago, but don't alert for any given user more than once an hour.

For more information about these alert types, see the sections below.

You can also create scheduled searches that fire off an action (such as an email with the results of the scheduled search) each time they are run, whether or not results are received. For example, you can use this method to set up a "failed logins" report that is sent out each day by email and which provides information on the failed logins over the previous day. For more information, see "Set up alert actions" in this manual.

Note: By default, only users with the Admin role can run and save real-time searches, schedule searches, or create alerts. In addition you cannot create saved searches unless your role permissions enable you to do so. For more information on managing roles, see "Add and edit roles with Splunk Web" in the Security Manual.

For a series of alert examples showing how you might design alerts for specific situations using both scheduled and real-time searches, see "Alert examples", in this manual.

Get started with alert creation using Splunk Web

If you run a search, like the results it's giving you, and decide that you'd like to base an alert on it, then click the Create button that appears above the search timeline.

4.3 alerting search action.png

Select Alert... to open the Create alert dialog on the Schedule step. Give the alert a Name and then select the alert Schedule. Use Schedule to determine the type of alert you want to configure. Your choice depends upon what you want to do with your alert.

4.3 alerting per-result schedule.png

You can choose:

Select the option that best describes the kind of alert you'd like to create.

Define per-result alerts

This documentation applies to the following versions of Splunk® Enterprise: 5.0, 5.0.1, 5.0.2, 5.0.3, 5.0.4, 5.0.5, 5.0.6, 5.0.7, 5.0.8, 5.0.9, 5.0.10, 5.0.11, 5.0.12, 5.0.13, 5.0.14, 5.0.15, 5.0.16, 5.0.17, 5.0.18

Was this documentation topic helpful?

Enter your email address, and someone from the documentation team will respond to you:

Please provide your comments here. Ask a question or make a suggestion.

You must be logged into splunk.com in order to post comments. Log in now.

Please try to keep this discussion focused on the content covered in this documentation topic. If you have a more general question about Splunk functionality or are experiencing a difficulty with Splunk, consider posting a question to Splunkbase Answers.

0 out of 1000 Characters