If you've been using Splunk's Search app for a while, you know how you can use Splunk's powerful search capabilities to learn all kinds of things about the machine data in your system. But this doesn't help you with the myriad of recurring situations that everyone in IT faces on a regular basis. You can't run searches yourself to find these events all of the time.
This is why we've designed Splunk to be the most flexible monitoring tool in your arsenal. You can configure a variety of alerting scenarios for your real-time and historical searches. You can have your historical searches run automatically on regular schedules, and you can set up both types of searches so they send alert messages to you and others when their results meet specific circumstances. You can base these alerts on a wide range of threshold and trend-based scenarios, including empty shopping carts, brute force firewall attacks, and server system errors.
In this manual you'll find:
- A summary of the three alert types and help with getting started with alert creation (see the following sections in this topic)
- Guides to the creation of three different kinds of alerts: per-result alerts, scheduled alerts, and rolling-window alerts.
- Help with setting up alert actions (such as email notifications).
- A variety of alerting examples.
- A guide to the Alert Manager, which enables you to manage recently triggered alerts.
- Details on setting up alerts via .conf files, including two conf. file alert setup examples:
The three alert categories
Splunk alerts are based on reports that run on a regular interval over a set historical time range or in real time (if the report is a real-time search). When the alerts trigger, different actions can take place, such as the sending of an email with the results of the triggering search to a predefined list of people.
Splunk enables you to design three broad categories of alerts:
|Type of alert||Base search is a...||Description||Alert examples|
|Alerts based on real-time searches that trigger every time the base search returns a result.||Real-time search (runs over all time)||Use this alert type if you need to know the moment a matching result comes in. This type is also useful if you need to design an alert for machine consumption (such as a workflow-oriented application). You can throttle these alerts to ensure that they don't trigger too frequently. Referred to as a "per-result alert."||
|Alerts based on historical searches that run on a regular schedule.||Historical search||This alert type triggers whenever a scheduled run of a historical search returns results that meet a particular condition that you have configured in the alert definition. Best for cases where immediate reaction to an alert is not a priority. You can use throttling to reduce the frequency of redundant alerts. Referred to as a "scheduled alert."||
|Alerts based on real-time searches that monitor events within a rolling time "window".||Real-time search||Use this alert type to monitor events in real time within a rolling time window of a width that you define, such as a minute, 10 minutes, or an hour. The alert triggers when its conditions are met by events as they pass through this window in real time. You can throttle these alerts to ensure that they don't trigger too frequently. Referred to as a "rolling-window alert."||
For more information about these alert types, see the sections below.
You can also create scheduled reports that fire off an action (such as an email with the results of the scheduled report) each time they run, whether or not they receive results. For example, you can use this method to set up a "failed logins" report that gets sent out each day by email and which provides information on the failed logins over the previous day. For more information, see "Schedule reports", in the Reporting Manual.
Note: When you use Splunk out-of-the-box, only users with the Admin role can run and save real-time searches, schedule searches, or create alerts. In addition, you cannot create reports unless your role permissions enable you to do so. For more information on managing roles, see "Add and edit roles with Splunk Web" in the Security Manual.
For a series of alert examples showing how you might design alerts for specific situations using both scheduled and real-time searches, see "Alert examples", in this manual.
Get started with creating alerts in Splunk Web
If you run a search, like the results it's giving you, and decide that you'd like to base an alert on it, then click the Save As button that appears above the search timeline.
1. Select Alert to open the Save As Alert dialog.
2. When the dialog opens, give the alert a Name and, optionally, a Description.
3. Select the Alert Type of the alert you want to configure: Real-time or Scheduled.
- Your choice depends upon what you want to do with your alert.
You can choose:
- "Real-time" to create a per-result alert.
- "Scheduled" to define a scheduled alert.
- To monitor alerts in real-time over a rolling window, select Real-time, then under trigger condition, click Number of Results. The dialog updates to let you define the window period. The "Define rolling-window alerts" topic has additional information.
Select the option that best describes the kind of alert you'd like to create.
Define per-result alerts
This documentation applies to the following versions of Splunk® Enterprise: 6.0, 6.0.1, 6.0.2, 6.0.3, 6.0.4, 6.0.5, 6.0.6, 6.0.7, 6.0.8, 6.0.9, 6.0.10, 6.0.11, 6.0.12, 6.0.13, 6.0.14, 6.0.15