Splunk® Enterprise

Alerting Manual

Download manual as PDF

Splunk Enterprise version 6.x is no longer supported as of October 23, 2019. See the Splunk Software Support Policy for details. For information about upgrading to a supported version, see How to upgrade Splunk Enterprise.
This documentation does not apply to the most recent version of Splunk. Click here for the latest version.
Download topic as PDF

About alerts

If you've been using Splunk's Search app for a while, you know how you can use Splunk's powerful search capabilities to learn all kinds of things about the machine data in your system. But this doesn't help you with the myriad of recurring situations that everyone in IT faces on a regular basis. You can't run searches yourself to find these events all of the time.

This is why we've designed Splunk to be the most flexible monitoring tool in your arsenal. You can configure a variety of alerting scenarios for your real-time and historical searches. You can have your historical searches run automatically on regular schedules, and you can set up both types of searches so they send alert messages to you and others when their results meet specific circumstances. You can base these alerts on a wide range of threshold and trend-based scenarios, including empty shopping carts, brute force firewall attacks, and server system errors.

In this manual you'll find:

The three alert categories

Splunk alerts are based on reports that run on a regular interval over a set historical time range or in real time (if the report is a real-time search). When the alerts trigger, different actions can take place, such as the sending of an email with the results of the triggering search to a predefined list of people.

Splunk enables you to design three broad categories of alerts:

Type of alert Base search is a... Description Alert examples
Alerts based on real-time searches that trigger every time the base search returns a result. Real-time search (runs over all time) Use this alert type if you need to know the moment a matching result comes in. This type is also useful if you need to design an alert for machine consumption (such as a workflow-oriented application). You can throttle these alerts to ensure that they don't trigger too frequently. Referred to as a "per-result alert."
  • Trigger an alert for every failed login attempt, but alert at most once an hour for any given username.
  • Trigger an alert when a "file system full" error occurs on any host, but only send notifications for any given host once per 30 minutes.
  • Trigger an alert when a CPU on a host sustains 100% utilization for an extended period of time, but only alert once every 5 minutes.
Alerts based on historical searches that run on a regular schedule. Historical search This alert type triggers whenever a scheduled run of a historical search returns results that meet a particular condition that you have configured in the alert definition. Best for cases where immediate reaction to an alert is not a priority. You can use throttling to reduce the frequency of redundant alerts. Referred to as a "scheduled alert."
  • Trigger an alert whenever the number of items sold in the previous day is less than 500.
  • Trigger an alert when the number of 404 errors in any 1 hour interval exceeds 100.
Alerts based on real-time searches that monitor events within a rolling time "window". Real-time search Use this alert type to monitor events in real time within a rolling time window of a width that you define, such as a minute, 10 minutes, or an hour. The alert triggers when its conditions are met by events as they pass through this window in real time. You can throttle these alerts to ensure that they don't trigger too frequently. Referred to as a "rolling-window alert."
  • Trigger an alert whenever there are three consecutive failed logins for a user between now and 10 minutes ago, but don't alert for any given user more than once an hour.
  • Trigger an alert when a host is unable to complete an hourly file transfer to another host within the last hour, but don't alert more than once an hour for any particular host.

For more information about these alert types, see the sections below.

You can also create scheduled reports that fire off an action (such as an email with the results of the scheduled report) each time they run, whether or not they receive results. For example, you can use this method to set up a "failed logins" report that gets sent out each day by email and which provides information on the failed logins over the previous day. For more information, see "Schedule reports", in the Reporting Manual.

Note: When you use Splunk out-of-the-box, only users with the Admin role can run and save real-time searches, schedule searches, or create alerts. In addition, you cannot create reports unless your role permissions enable you to do so. For more information on managing roles, see "Add and edit roles with Splunk Web" in the Security Manual.

For a series of alert examples showing how you might design alerts for specific situations using both scheduled and real-time searches, see "Alert examples", in this manual.

Get started with creating alerts in Splunk Web

If you run a search, like the results it's giving you, and decide that you'd like to base an alert on it, then click the Save As button that appears above the search timeline.

Bubbles saveasalert realtime.png

1. Select Alert to open the Save As Alert dialog.

2. When the dialog opens, give the alert a Name and, optionally, a Description.

3. Select the Alert Type of the alert you want to configure: Real-time or Scheduled.

Your choice depends upon what you want to do with your alert.

60 saveasalert realtime.png

You can choose:

  • "Real-time" to create a per-result alert.
  • "Scheduled" to define a scheduled alert.
  • To monitor alerts in real-time over a rolling window, select Real-time, then under trigger condition, click Number of Results. The dialog updates to let you define the window period. The "Define rolling-window alerts" topic has additional information.

Select the option that best describes the kind of alert you'd like to create.

Last modified on 28 June, 2016
Define per-result alerts

This documentation applies to the following versions of Splunk® Enterprise: 6.0, 6.0.1, 6.0.2, 6.0.3, 6.0.4, 6.0.5, 6.0.6, 6.0.7, 6.0.8, 6.0.9, 6.0.10, 6.0.11, 6.0.12, 6.0.13, 6.0.14, 6.0.15

Was this documentation topic helpful?

Enter your email address, and someone from the documentation team will respond to you:

Please provide your comments here. Ask a question or make a suggestion.

You must be logged into splunk.com in order to post comments. Log in now.

Please try to keep this discussion focused on the content covered in this documentation topic. If you have a more general question about Splunk functionality or are experiencing a difficulty with Splunk, consider posting a question to Splunkbase Answers.

0 out of 1000 Characters