If you've been using Splunk's Search app for a while, you know how you can use its powerful search capabilities to learn all kinds of things about the machine data in your system. But this doesn't help you with the myriad of recurring situations that everyone in IT is faced with on a regular basis. You can't be running searches yourself to find these events all of the time.
This is why we've designed Splunk Enterprise to be the most flexible monitoring tool in your arsenal. You can configure a variety of alerting scenarios for your real-time and historical searches. You can have your historical searches run automatically on regular schedules, and you can set up both types of searches so they send alert messages to you and others when their results meet specific circumstances. You can base these alerts on a wide range of threshold and trend-based scenarios, including empty shopping carts, brute force firewall attacks, and server system errors.
In this manual you'll find:
- A summary of the three alert types and help with getting started with alert creation (see the following sections in this topic)
- Guides to the creation of three different kinds of alerts: per-result alerts, scheduled alerts, and rolling-window alerts.
- Help with setting up alert actions (such as email notifications).
- A variety of alerting examples.
- A guide to the Alert Manager, which enables you to manage recently triggered alerts.
- Instruction for setting up scheduled searches--searches that run on a regular interval and which trigger an alert action (such as the sending of an email with search results) each time they run. Scheduled searches are also used for summary indexing.
- Details on setting up alerts via .conf files, including two conf. file alert setup examples:
The three alert categories
Splunk alerts are based on saved searches that run on a regular interval over a set historical time range or in real time (if the saved search is a real-time search). When they are triggered, different actions can take place, such as the sending of an email with the results of the triggering search to a predefined list of people.
There are three broad categories of alerts:
|Type of alert||Base search is a...||Description||Alert examples|
|Alerts based on real-time searches that are triggered every time the base search returns a result.||Real-time search (runs over all time)||Use this alert type if you need to know the moment a matching result comes in. Useful if you need to design an alert for machine consumption (such as a workflow-oriented application). You can also throttle these alerts to ensure that they aren't triggered too frequently. Referred to as a "per-result alert."||
|Alerts based on historical searches that run on a regular schedule.||Historical search||This alert type triggers whenever a scheduled run of a historical search returns results that meet a particular condition that you have configured in the alert definition. Best for cases where immediate reaction to an alert is not a priority. You can use throttling to reduce the frequency of redundant alerts. Referred to as a "scheduled alert."||
|Alerts based on real-time searches that monitor events within a rolling time "window".||Real-time search||Use this alert type to monitor events in real time within a rolling time window of a width that you define, such as a minute, 10 minutes, or an hour. The alert is triggered when its conditions are met by events as they pass through this window in real time. You can throttle these alerts to ensure that they aren't triggered too frequently. Referred to as a "rolling-window alert."||
For more information about these alert types, see the sections below.
You can also create scheduled searches that fire off an action (such as an email with the results of the scheduled search) each time they are run, whether or not results are received. For example, you can use this method to set up a "failed logins" report that is sent out each day by email and which provides information on the failed logins over the previous day. For more information, see "Set up alert actions" in this manual.
Note: By default, only users with the Admin role can run and save real-time searches, schedule searches, or create alerts. In addition you cannot create saved searches unless your role permissions enable you to do so. For more information on managing roles, see "Add and edit roles with Splunk Web" in the Security Manual.
For a series of alert examples showing how you might design alerts for specific situations using both scheduled and real-time searches, see "Alert examples", in this manual.
Get started with alert creation using Splunk Web
If you run a search, like the results it's giving you, and decide that you'd like to base an alert on it, then click the Create button that appears above the search timeline.
Select Alert... to open the Create alert dialog on the Schedule step. Give the alert a Name and then select the alert Schedule. Use Schedule to determine the type of alert you want to configure. Your choice depends upon what you want to do with your alert.
You can choose:
- Trigger in real-time whenever a result matches to create a per-result alert.
- Run on a schedule once every... to define a scheduled alert.
- Monitor in real-time over a rolling window of... to set up a rolling-window alert.
Select the option that best describes the kind of alert you'd like to create.
Define per-result alerts
This documentation applies to the following versions of Splunk® Enterprise: 5.0, 5.0.1, 5.0.2, 5.0.3, 5.0.4, 5.0.5, 5.0.6, 5.0.7, 5.0.8, 5.0.9, 5.0.10, 5.0.11, 5.0.12, 5.0.13, 5.0.14, 5.0.15, 5.0.16, 5.0.17, 5.0.18