Splunk® Enterprise

Alerting Manual

Download manual as PDF

This documentation does not apply to the most recent version of Splunk. Click here for the latest version.
Download topic as PDF

Create scheduled alerts

A scheduled alert evaluates the results of a historical search that runs over a specified time range on a regular schedule. The alert fires when it encounters the trigger condition.

For example, you can create a scheduled alert to monitor online sales. The search runs daily at midnight and triggers when the sum of the sales of a specific item is below 500 for the previous day. When the alert triggers, it sends an email to the appropriate administrators monitoring sales.

  1. From the Search Page, create the following search. Select Last 24 Hours for the time range:

    index=_internal (log_level=ERROR OR log_level=WARN* OR log_level=FATAL OR log_level=CRITICAL) | stats count as log_events

  2. Select Save As > Alert
  3. In the Save As Alert dialog box, specify the following:

    • Title: Server Errors Last 24 hours
    • Alert Type: Scheduled
    • Time Range: Run Every Day
    • Schedule At: 0:00
    • Trigger Condition: Number of Results
    • Trigger if number of results: is Greater than 5
    Alert create scheduled.png
  4. Click Next.
  5. In the Save As Alert dialog box, specify the following:

    • Enable Actions: List in Triggered Alerts
    • When triggered, execute actions: Once
    See Set up alert actions for information on other actions.

    Alert create scheduled2.png
  6. Click Save.
  7. Click Continue Editing.

Use cron notation for scheduled alerts

When scheduling an alert, you can use cron notation for customized schedules. When specifying a cron schedule, only five cron parameters are available, not six. The sixth parameter for year, common in other forms of cron notation, is not available.

The following cron parameters:

* * * * *

correspond to:

minute hour day month day-of-week

Following are some cron examples:

*/5 * * * *       Every 5 minutes.
*/30 * * * *      Every 30 minutes.
0 */12 * * *      Every 12 hours, on the hour.
*/20  * * * 1-5   Every 20 minutes, Monday through Friday.
0 9 1-7 * 1       First Monday of each month, at 9am.

When you select Run on Cron Schedule for the time range of a scheduled alert, enter the earliest and latest parameters for a search. What you enter overrides the time range you set when you first ran the search.

To avoid overlaps or gaps, the execution schedule should match the search time range. For example, to run a search every 20 minutes the search's time range should also be 20 minutes (-20m).

Alert cron schedule.png

Best practices for scheduled alerts

This section discusses some best practices for scheduled alerts.

Coordinate an alert's schedule with the search time range

Coordinating the alert's schedule with the search time range prevents situations where event data is evaluated twice by the search. This can happen if the search time range exceeds the search schedule, resulting in overlapping event data sets.

In cases where the search time range is shorter than the time range for the scheduled alert, an event might never be evaluated.

Schedule alerts with at least 60 seconds of delay

This practice is important in distributed search deployments where event data might not reach the indexer precisely at the moment when it is generated. A delay ensures that you are counting all events, not just the events that were quickest to get indexed.

Best practices example

This example shows how to configure an alert that builds 30 minutes of delay into the alert schedule. Both the search time range and the alert schedule span one hour, so there are no event data overlaps or gaps.

The alert runs every hour at the half hour. It collects an hour's worth of event data, beginning an hour and a half before the search runs. When the scheduled search kicks off at a designated time, such as 3:30 pm, it collects the event data that was indexed from 2:00 pm to 3:00 pm.

  1. From the Search Page, create a search and select Save As > Alert.
  2. In the Save As Alert dialog, specify the following to schedule the alert:

    • Title: Alert Example (30 Minute Delay)
    • Alert Type: Scheduled
    • Time Range: Run on Cron Schedule
    • Earliest: -90m
    • Latest: -30
      Earliest and Latest values set the time that the search covers to a period that begins 90 minutes before the search launch time, ending 30 minutes before the search launch time.
    • Cron Expression: 30 * * * *
      The alert runs every hour on the half hour
    Alert 30 min delay schedule.png
  3. Click Next and continue defining actions for the alert.

Manage the priority of concurrently scheduled searches

Depending on your Splunk Enterprise deployment, you might be able to run only one scheduled search at a time. In this case, when you schedule multiple searches to run at approximately the same time, the search scheduler ensures that all scheduled searches run consecutively for the period of time over which they gather data.

However, you might have cases where you need certain searches to run ahead of others. This is to ensure that the searches obtain current data or to ensure that there are no gaps in data collection.

You can configure the priority of scheduled searches in the savedsearches.conf configuration file. See "Configure the priority of scheduled reports" in the Reporting Manual.

Set up triggering conditions for a scheduled alert

Trigger conditions apply to two types of conditional alerts:

  • Basic conditional alert
  • Advanced conditional alert

Set the triggering conditions when you set values for the Trigger condition field in the Save As Alert dialog box, as described in the following subtopics.

Basic conditional alert

A basic conditional alert triggers when the number of results of a scheduled search meet, exceed, or are less than a specified numerical value. When you create the alert, you can specify the following conditions:

  • Number of results
  • Number of hosts
  • Number of sources

The alert triggers when the number of hosts in the results rises by a count of more than 12.

  1. From the Search Page, create a search and select Save As > Alert.
  2. In the Save As Alert dialog box, specify the following fields to schedule the alert:

    • Title: Alert Example (Basic Conditional)
    • Alert Type: Scheduled
      You can also select Real Time for a basic conditional search.
    • Time Range and Schedule: Select any time range and schedule.
    • Trigger Condition: Number of Hosts
      You can also select Number of Results or Number of Sources
    • Trigger if number of results: Select a comparison operator and trigger value.
    Alert basic conditional schedule.png
  3. Click Next and continue defining actions for the alert.

Basic conditional alert for rolling-window alerts

The behavior for basic conditional alerts differs slightly for a rolling-window alert. The alert triggers when the set condition occurs within the rolling time window of the search.

For example, a rolling-window alert that triggers when a time window of 60 seconds has five or more results. If the real-time search returns one result and then four more results five minutes later, the alert does not trigger. The alert does trigger If the search returns five results within a single 60-second span.

Advanced conditional alert

An advanced conditional alert uses a secondary, custom conditional search to evaluate the results of a scheduled or real-time search. The alert triggers when the custom search returns any number of results. If the alerting conditions are not met, then the custom conditional search returns zero results.

A secondary conditional search can help reduce the incidence of false positive alerts.

In the following example, the alert triggers when there are 10 or more log level events that are not INFO. When the alert triggers, it sends an email with the results of the search. The search results detail the count for each log level.

  1. From the Search Page, create the following search. Specify Last 7 days for the time period.

    index=_internal (log_level=ERROR OR log_level=FATAL OR log_level=CRITICAL) | stats count by log_level

  2. Select Save As > Alert.
  3. In the Save As Alert dialog box, specify the following fields to schedule the alert:

    • Title: Alert Example (Advanced Conditional)
    • Alert Type: Scheduled
      You can also select Real Time for an advanced conditional search.
    • Time Range and Schedule: Select any time range and schedule.
    • Trigger Condition: Custom
    • Custom condition: search count > 10
    Alert advanced conditional schedule.png
  4. Click Next.
  5. Define an action that sends an email that includes the results of the search.
    When you configure a Send Email action that includes search results, the email contains the results of the original base search. It does not include the results of the custom search.

It might appear that you can get the same results if you specify instead, the following search for the base search of a basic conditional search:

log_level=ERROR OR log_level=FATAL OR log_level=CRITICAL) | stats count by log_level | search count > 10

However, a basic conditional alert based on this search provides different results. The search results contain only log level values that are greater than 10. The results from the advanced conditional search details the count for all log levels, but triggers only when the log levels are greater than 10.

Advanced conditional alert for rolling-window alerts

The behavior for advanced conditional alerts differs slightly for a rolling-window alert, which runs in real-time. For a rolling-window alert, the alert triggers when the set condition occurs within the rolling time window of the search.

For the previous example, you can design a rolling-window alert with the same base search and get similar results with the custom condition search. Set the rolling window to a 10 minutes time span. When the real-time search returns 10 log level entries within the 10 minute time span, the alert triggers.

For more examples of scheduled alerts, see "Alert examples," in this manual.

PREVIOUS
Create per-result alerts
  NEXT
Create rolling-window alerts

This documentation applies to the following versions of Splunk® Enterprise: 6.2.0, 6.2.1, 6.2.2, 6.2.3, 6.2.4, 6.2.5, 6.2.6, 6.2.7, 6.2.8, 6.2.9, 6.2.10, 6.2.11, 6.2.12, 6.2.13, 6.2.14, 6.2.15


Was this documentation topic helpful?

Enter your email address, and someone from the documentation team will respond to you:

Please provide your comments here. Ask a question or make a suggestion.

You must be logged into splunk.com in order to post comments. Log in now.

Please try to keep this discussion focused on the content covered in this documentation topic. If you have a more general question about Splunk functionality or are experiencing a difficulty with Splunk, consider posting a question to Splunkbase Answers.

0 out of 1000 Characters