Splunk® Enterprise

Alerting Manual

Download manual as PDF

Splunk Enterprise version 5.0 reached its End of Life on December 1, 2017. Please see the migration information.
This documentation does not apply to the most recent version of Splunk. Click here for the latest version.
Download topic as PDF

Define scheduled alerts

Note: This topic explains how to define scheduled alerts, one of three types of alerts that Splunk provides. For an overview of the alert types, and more information about getting started with alert creation, go to About alerts, in this manual.

Scheduled alerts are the second most common alert type. If you want to set up an alert that evaluates the results of a historical search that runs over a set range of time on a regular schedule, you would select a Schedule of Run on a schedule once every... on the Schedule step of the Create alert dialog box. Then you'd select an interval for the schedule and define the alert triggering conditions.

This schedule can be of any duration that you define. For example, say you handle operations for an online store, and you'd like the store administrators to receive an alert via email if the total sum of items sold through the store in the previous day drops below a threshold of 500 items. They don't need to know the moment this happens, they just want to be alerted on a day by day basis.

4.3 alerting scheduled schedule.png

To meet these requirements, you'd create an alert based on a search that runs on a regular schedule. You would schedule the search to run every day at midnight, and configure it to be triggered whenever the result returned by a scheduled run of the search--the sum of items sold over the past day--is a number below 500. And you'd set up an email alert action for the search. This ensures that when the search is triggered, Splunk sends out an email to every email address in the alert action definition, which in this case would be the set of store administrators. Now your administrators will be informed when any midnight run of the search finds that less than 500 items were purchased in the previous day.

Schedule the alert

You can schedule a search for a scheduled alert on the Schedule step of the Create Alert dialog. After you select Run on a schedule once every... for the Schedule field, select the schedule interval from the list that appears. By default the schedule will be set to Hour but you can change this to Day, Week, Month, or select Cron schedule to set up a more complicated interval that you define using standard cron notation.

4.3 alerting schedule by day.png

Note: Splunk only uses 5 parameters for cron notation, not 6. The parameters (* * * * *) correspond to minute hour day month day-of-week. Splunk does not use the 6th parameter for year, common in other forms of cron notation.

Here are some cron examples:

*/5 * * * *       : Every 5 minutes
*/30 * * * *      : Every 30 minutes
0 */12 * * *      : Every 12 hours, on the hour
*/20  * * * 1-5   : Every 20 minutes, Monday through Friday
0 9 1-7 * 1       : First Monday of each month, at 9am.

If you set choose Cron schedule, you also need to enter a Search time range over which you want to run the search. What you enter here will override the time range you set when you first designed and ran the search. It is recommended that the execution schedule matches the search time range so that no overlaps or gaps occurs - for example if you choose to run a search every 20 minutes then it is recommended that the search's time range be 20 minutes as well (-20m).

Alert scheduling best practices

  • Coordinate the alert's search schedule with the search time range. This prevents situations where event data is accidentally evaluated twice by the search (because the search time range exceeds the search schedule, resulting in overlapping event data sets), or not evaluated at all (because the search time range is shorter than the search schedule).
  • Schedule your alerting searches with at least 60 seconds of delay. This practice is especially important in distributed search Splunk implementations where event data may not reach the indexer precisely at the moment when it is generated. A delay ensures that you are counting all of your events, not just the ones that were quickest to get indexed.

The following example sets up a search that runs every hour at the half hour, and which collects an hour's worth of event data, beginning an hour and a half before the search is actually run. This means that when the scheduled search kicks off at 3:30pm, it is collecting the event data that Splunk indexed from 2:00pm to 3:00pm. In other words, this configuration builds 30 minutes of delay into the search schedule. However both the search time range and the search schedule span 1 hour, so there will be no event data overlaps or gaps.

  • Select Run on a schedule once every... for Schedule
  • Select Cron schedule from the interval list.
  • In the cron notation field, enter 30 * * * * to have your search run every hour on the half hour.
  • Set the time range of the search using relative time modifier syntax. Enter an Earliest time value of -90m and a Latest time value of -30m. This means that each time the search runs it covers a period that begins 90 minutes before the search launch time, and ends 30 minutes before the search launch time.

For more information about the relative time modifier syntax for search time range definition, see "Specify time modifiers in your search" in the Search Manual.

Manage the priority of concurrently scheduled searches

Depending on how you have your Splunk implementation set up, you may only be able to run one scheduled search at a time. Under this restriction, when you schedule multiple searches to run at approximately the same time, Splunk's search scheduler works to ensure that all of your scheduled searches get run consecutively for the period of time over which they are supposed to gather data. However, there are cases where you may need to have certain searches run ahead of others in order to ensure that current data is obtained, or to ensure that gaps in data collection do not occur (depending on your needs).

You can configure the priority of scheduled searches through edits to savedsearches.conf. For more information about this feature, see "Configure the priority of scheduled searches" in the Knowledge Manager manual.

Set up triggering conditions for a scheduled alert

An alert based on a scheduled search is triggered when the results returned by a scheduled run of the base search meet specific conditions such as passing a numerical threshold.

These triggering conditions break scheduled alerts into two subcategories: basic conditional scheduled alerts and advanced conditional scheduled alerts. You set these triggering conditions when you set values for the Trigger if field on the Schedule step of the Create Alert dialog.

A basic conditional alert is triggered when the number of events in the results of a scheduled search meet a simple alerting condition (for example, a basic conditional alert can be triggered when the number of events returned by its search are greater than, less than, or equal to a number that you provide).

Note: If you define or edit your alert at Manager > Searches and Reports, the Condition field enables you to additionally set up basic conditional alerts that monitor numbers of hosts or sources amongst the events reviewed by the search.

An advanced conditional alert uses a secondary "conditional" search to evaluate the results of the scheduled or real-time search. With this setup, the alert is triggered when the secondary search returns any results.

Define a basic conditional alert

On the Schedule step of the Create Alert dialog, follow this procedure to define a basic conditional alert that notifies you when a simple alerting condition is met by the number of events returned by the search. Schedule must be set to Run on a schedule once every... in order for you to see the fields described in this procedure.

4.3 alerting basic conditional-1.png

1. Set Trigger if to Number of results. This is the basic condition that triggers the result.

Note: If you are defining or updating your alert at Manager > Searches and reports the Condition field additionally enables you to select Number of hosts and Number of sources.

2. Choose a comparison operation from the list below the Trigger if field. You can select Is greater than, Is less than, Equals to, or Is not equal to depending on how you want the alerting condition to work.

Note: If you are defining or updating your alert at Manager > Searches and reports you can additionally select rises by and drops by for comparison operations. These operations compare the results of the current run of the search against the results returned by the last run of the search. Rises by and drops by are not available for alerts based on real-time searches.)

3. In the field adjacent to the comparison operation list, enter an integer to complete the basic conditional alert. This integer represents a number of events.

The alert is triggered if the results for a scheduled run of the search meet or exceed the set condition. For example, you can set up an alert for a scheduled search that sends out a notification if the number of results returned by the search is less than a threshold of 10 results.

Note: Basic conditional alerts work differently for scheduled searches (which use a historical, scheduled search) and rolling-window alerts (which use a real-time search with a rolling time window of a specified width).

When you define a rolling-window alert as a basic conditional alert, the alert is triggered when the set condition occurs within the rolling time window of the search. For example, you could have a rolling-window alert that triggers the moment that the rolling 60 second window for the search has 5 or more results all at the same time.

Just to be clear, this means that if the real-time search returns one result and then four more results five minutes later, the alert is not triggered. But if all five results are returned within a single 60-second span of time, the alert is triggered.

Define an advanced conditional alert

In an advanced conditional alert, you define a secondary conditional search that Splunk applies to the results of the scheduled search. If the secondary search returns any results, Splunk triggers the alert. This means that you need to use a secondary search that returns zero results when the alerting conditions are unmet.

By basing your alert conditions on the result of a secondary conditional search, you can define specific conditions for triggering alerts and reduce the incidence of false positive alerts.

4.3 alerting advanced conditional.png

Follow this procedure to define an advanced conditional alert:

1. In the Trigger if list, select Custom condition is met. The Custom condition field appears.

2. Enter your conditional search in the Custom condition field.

3. Complete your alert definition by defining alert actions, throttling rules, and sharing (see below).

Every time the base search runs on its schedule, the conditional search runs against the output of that search. Splunk triggers the alert if the conditional search returns one or more results.

Note: If you have set up a Send email action for an alert that uses a conditional search, and you've arranged for results to be included in the email, be aware that Splunk will include the results of the original base search. It does not send the results of the secondary conditional search.

Advanced conditional alert example

Lets say you're setting up an alert for the following scheduled search, which is scheduled to run every 10 minutes:

failed password | stats count by user

This search returns the number of incorrect password entries associated with each user name.

What you want to do is arrange to have Splunk trigger the alert when the scheduled search finds more than 10 password failures for any given user. When the alert is triggered, Splunk sends an email containing the results of the triggering search to interested parties.

Now, it seems like you could simply append | search count > 10 to the original scheduled search:

failed password | stats count by user | search count > 10

Unfortunately, if you create a basic conditional alert based on this search, where an alert is triggered when the number of results returned by the base search is greater than 10, you won't get the behavior you desire. This is because this new search only returns user names that are associated with 10+ failed password entries--the actual count values are left out. When the alert is triggered and the results are emailed to stakeholders, you want the recipients to have a listing that matches each user name to the precise number of failed password attempts that it is associated with.

What you want to do is set Condition to If custom condition is met and then place search count > 10 in the Custom condition search field (while removing it from the base search). This conditional search runs against the results of the original scheduled search (failed password | stats count by user). With this, the alert is triggered only when the custom condition is met--when there are 1 or more user names associated with 10 failed password entries. But when it is triggered, the results of the original search--the list of user names and their failed password counts--is sent to stakeholders via email.

Note: Advanced conditional alerts work slightly differently when you are designing them for rolling-window alerts, which run in real-time rather than on a schedule. In the case of the above example, you could design a rolling-window alert with the same base search and get similar results with the custom condition search as long as the rolling window was set to be 10 minutes wide. As soon as the real-time search returns 10 failed password entries for the same user within that 10-minute span of time, the alert is triggered.

For more examples of scheduled alerts, see "Alert examples," in this manual.

Enable actions for an alert based on a scheduled search

On the Actions step for a scheduled alert, you can enable one or more alert actions. These actions are set off whenever the alert is triggered.

There are three kinds of alert actions that you can enable through the Create alert dialog. For Enable actions you can select any combination of:

  • Send email - Send an email to a list of recipients that you define. You can opt to have this email contain the results of the triggering search job.
  • Run a script - Run a shell script that can perform some other action, such as the sending of an SNMP trap notification or the calling of an API. You determine which script is run.
  • Show triggered alerts in Alert manager - Have triggered alerts display in the Alert Manager with a severity level that you define. The severity level is non-functional and is for informational purposes only. (Note: In Manager > Searches and Reports, to have trigger records for an alert display in the Alert Manager, you enable the Tracking alert action.)

You can enable any combination of these alert actions for an individual alert.

4.3 alerting enable actions.png

Note: You can also arrange to have Splunk post the result of the triggered alert to an RSS feed. To enable this option, go to Manager > Searches and Reports and click the name of the search that the alert is based upon. Then, in the Alert actions section, click Enable for Add to RSS.

Important: Before enabling actions, read "Set up alert actions," in this manual. This topic discusses the various alert actions at length and provides important information about their setup. It also discusses options that are only available via the Searches and reports page in Manager, such as the ability to send reports with alert emails in PDF format, RSS feed notification, and summary indexing enablement.

Determine how often actions are executed when scheduled alerts are triggered

When you are setting up an alert based on a scheduled, historical search, you use the last two settings on the Actions step--Execute actions on and Throttling--to determine how often Splunk executes actions after an alert is triggered.

Execute actions on enables you to say that once an alert is triggered, the alert actions are executed for All results returned by the triggering search, or Each result returned by the triggering search. In other words, you can set it up so that actions are triggered only once per search, or so that actions are triggered multiple times per search: once for each result returned.

After you choose that setting, you can choose whether or not those actions should be throttled in some manner.

If you select All results, you can say that later alert actions should be suppressed for a specific number of seconds, minutes, or hours.

4.3 alerting scheduled allresultsthrottle.png

For example, say you have an alert based on a scheduled search that runs every half hour.

  • It has All results selected and throttling set to suppress actions for two hours, and on the Actions step it has Send email and Show triggered results in Alert manager set as alert actions.
  • It runs on its schedule, and the alerting conditions are met, so it is triggered.
  • Alert actions are executed once for all results returned by the alert, per the All results setting. Splunk sends alert emails out to the listed recipients and shows the triggered alert on the Alert manager page.
  • Then the alert runs again on its schedule, a half-hour later, and it's triggered again. But because the alert's throttling controls are set to suppress alert actions for two hours, nothing happens. No more alert actions can be executed until two hours (or three more runs of the scheduled search upon which the alert is based) have passed.

If the alert is triggered again after the two hours are up, the alert actions are executed again, same as they were last time. And then they are suppressed again for another two hours.

If you select Each result, the throttling rules are different, because when the alert is triggered, multiple actions can be executed, one for each result returned by the search. You can throttle action execution for results that share a particular field value.

4.3 alerting scheduled eachresultthrottle.png

Here's an example of a scheduled alert that performs actions for every result once it's triggered:

Say you're a system administrator who is responsible for a number of network servers, and when these servers experience excessive amounts of errors over a 24-hour period, you'd like to run a system check on them to get more information. Servers are identified in your logs with the servername field. Here's what you do:

1. Start by designing a script that performs a system check of a network server when it's given a value for the servername field, and which subsequently returns the results of that check back to you.

2. Then design a search that returns the servername values of machines that have experienced 5 or more errors over the past 24 hours, one value per result.

3. Next, open the Create Alert dialog for this search. In the Schedule step, use cron notation to schedule the search so it runs once each day at midnight. Because its defined time range is the past 24 hours, this means it'll return results for the previous day.

4. Set the search up so that it's triggered whenever more than 0 results are returned.

5. On the Actions step, enable the Run a script action and assign your system check script to it.

6. Finally, set up the alert so that when it's triggered, it executes the "run script" action for each result received.

This is a case where you'd likely keep throttling off, since your search is set up to return error sums by servername, and only for those servers that experience > 5 errors in the 24 hour time range of the search. You won't have to worry about getting multiple results with the same servername value, and there probably isn't much value in suppressing events when the search is run on a daily basis.

Share scheduled alerts with others

On the Sharing step for an alert based on a scheduled search, you can determine how the alert is shared if you have a role that gives you Write access to the knowledge objects in your app (such as the Power or Admin roles).

Sharing rules are the same for all alert types: you can opt to keep the search private, or you can share the alert as read-only to all users of the app you're currently using. For the latter choice, "read-only" means that other users of your current app can see and use the alert, but they can't update its definition via Manager > Searches and reports.

4.3 alerting sharing.png

If you have edit permissions for the alert, you can find additional permission settings in Manager > Searches and reports. For more information about managing permissions for Splunk knowledge objects (such as alert-enabled searches) read "Manage knowledge object permissions" in the Knowledge Manager Manual.

PREVIOUS
Define per-result alerts
  NEXT
Define rolling-window alerts

This documentation applies to the following versions of Splunk® Enterprise: 5.0, 5.0.1, 5.0.2, 5.0.3, 5.0.4, 5.0.5, 5.0.6, 5.0.7, 5.0.8, 5.0.9, 5.0.10, 5.0.11, 5.0.12, 5.0.13, 5.0.14, 5.0.15, 5.0.16, 5.0.17, 5.0.18


Was this documentation topic helpful?

Enter your email address, and someone from the documentation team will respond to you:

Please provide your comments here. Ask a question or make a suggestion.

You must be logged into splunk.com in order to post comments. Log in now.

Please try to keep this discussion focused on the content covered in this documentation topic. If you have a more general question about Splunk functionality or are experiencing a difficulty with Splunk, consider posting a question to Splunkbase Answers.

0 out of 1000 Characters