Splunk® Enterprise

Alerting Manual

Download manual as PDF

This documentation does not apply to the most recent version of Splunk. Click here for the latest version.
Download topic as PDF

Create scheduled alerts

A scheduled alert evaluates the results of a historical search that runs over a specified time range on a regular schedule. The alert fires when it encounters the trigger condition.

For example, you can create a scheduled alert to monitor online sales. The search runs daily at midnight and triggers when the sum of the sales of a specific item is below 500 for the previous day. When the alert triggers, it sends an email to the appropriate administrators monitoring sales.

  1. From the Search Page, create the following search. Select Last 24 Hours for the time range:

    index=_internal (log_level=ERROR OR log_level=WARN* OR log_level=FATAL OR log_level=CRITICAL) | stats count as log_events

  2. Select Save As > Alert
    The Save As Alert dialog box opens.
  3. Specify Settings:

    • Title: Server Errors Last 24 hours
    • Alert Type: Scheduled
    • Time Range: Run Every Day
    • Schedule At: 0:00
    • Trigger Condition: Number of Results
    • Trigger if number of results: is Greater than 5
  4. Specify Trigger Conditions:

    • Trigger alert when: Number of Results is Greater than 5
    • Trigger it: Once
  5. Specify Trigger Actions:

    • Add Actions: List in Triggered Alerts
    See Set up alert actions for information on other actions.

  6. Click Save.


Use cron notation for scheduled alerts

When scheduling an alert, you can use cron notation for customized schedules. When specifying a cron schedule, only five cron parameters are available, not six. The sixth parameter for year, common in other forms of cron notation, is not available.

The following cron parameters:

* * * * *

correspond to:

minute hour day month day-of-week

Following are some cron examples:

*/5 * * * *       Every 5 minutes.
*/30 * * * *      Every 30 minutes.
0 */12 * * *      Every 12 hours, on the hour.
*/20  * * * 1-5   Every 20 minutes, Monday through Friday.
0 9 1-7 * 1       First Monday of each month, at 9am.

When you select Run on Cron Schedule for the time range of a scheduled alert, enter the earliest and latest parameters for a search. What you enter overrides the time range you set when you first ran the search.

To avoid overlaps or gaps, the execution schedule should match the search time range. For example, to run a search every 20 minutes the search's time range should also be 20 minutes (-20m). Alert cron schedule.png

Manage the priority of concurrently scheduled searches

Depending on your Splunk Enterprise deployment, you might be able to run only one scheduled search at a time. In this case, when you schedule multiple searches to run at approximately the same time, the search scheduler ensures that all scheduled searches run consecutively for the period of time over which they gather data.

However, you might have cases where you need certain searches to run ahead of others. This is to ensure that the searches obtain current data or to ensure that there are no gaps in data collection.

You can configure the priority of scheduled searches in the savedsearches.conf configuration file. See "Configure the priority of scheduled reports" in the Reporting Manual.

Best practices for scheduled alerts

This section discusses some best practices for scheduled alerts.

Coordinate an alert's schedule with the search time range

Coordinating the alert's schedule with the search time range prevents situations where event data is evaluated twice by the search. This can happen if the search time range exceeds the search schedule, resulting in overlapping event data sets.

In cases where the search time range is shorter than the time range for the scheduled alert, an event might never be evaluated.

Schedule alerts with at least 60 seconds of delay

This practice is important in distributed search deployments where event data might not reach the indexer precisely at the moment when it is generated. A delay ensures that you are counting all events, not just the events that were quickest to get indexed.

Best practices example

This example shows how to configure an alert that builds 30 minutes of delay into the alert schedule. Both the search time range and the alert schedule span one hour, so there are no event data overlaps or gaps.

The alert runs every hour at the half hour. It collects an hour's worth of event data, beginning an hour and a half before the search runs. When the scheduled search kicks off at a designated time, such as 3:30 pm, it collects the event data that was indexed from 2:00 pm to 3:00 pm.

  1. From the Search Page, create a search and select Save As > Alert.
  2. In the Save As Alert dialog, specify the following to schedule the alert:

    • Title: Alert Example (30 Minute Delay)
    • Alert Type: Scheduled
    • Time Range: Run on Cron Schedule
    • Earliest: -90m
    • Latest: -30m
      Earliest and Latest values set the time that the search covers to a period that begins 90 minutes before the search launch time, ending 30 minutes before the search launch time.
    • Cron Expression: 30 * * * *
      The alert runs every hour on the half hour
    Alert 30 min delay schedule.png
  3. Continue defining actions for the alert.

Set up triggering conditions for a scheduled alert

Trigger conditions apply to two types of conditional alerts:

  • Basic conditional alert
  • Advanced conditional alert

Set the triggering conditions when you set values for the Trigger condition field in the Save As Alert dialog box, as described in the following subtopics.

Basic conditional alert

A basic conditional alert triggers when the number of results of a scheduled search meet, exceed, or are less than a specified numerical value. When you create the alert, you can specify the following conditions:

  • Number of results
  • Number of hosts
  • Number of sources

The alert triggers when the number of hosts in the results rises by a count of more than 12.

  1. From the Search Page, create a search and select Save As > Alert.
  2. In the Save As Alert dialog box, specify the following fields to schedule the alert:

    • Title: Alert Example (Basic Conditional)
    • Alert Type: Scheduled
      You can also select Real Time for a basic conditional search.
    • Time Range and Schedule: Select any time range and schedule.
    • Trigger Condition: Number of Hosts
      You can also select Number of Results or Number of Sources
    • Trigger if number of results: Select a comparison operator and trigger value.
    Alert basic conditional schedule.png
  3. Continue defining actions for the alert.

Basic conditional alert for rolling-window alerts

The behavior for basic conditional alerts differs slightly for a rolling-window alert. The alert triggers when the set condition occurs within the rolling time window of the search.

For example, a rolling-window alert that triggers when a time window of 60 seconds has five or more results. If the real-time search returns one result and then four more results five minutes later, the alert does not trigger. The alert does trigger If the search returns five results within a single 60-second span.

Advanced conditional alert

An advanced conditional alert uses a secondary, custom conditional search to evaluate the results of a scheduled or real-time search. The alert triggers when the custom search returns any number of results. If the alerting conditions are not met, then the custom conditional search returns zero results.

A secondary conditional search can help reduce the incidence of false positive alerts.

In the following example, the alert triggers when there are 10 or more log level events that are not INFO. When the alert triggers, it sends an email with the results of the search. The search results detail the count for each log level.

  1. From the Search Page, create the following search. Specify Last 7 days for the time period.

    index=_internal (log_level=ERROR OR log_level=FATAL OR log_level=CRITICAL) | stats count by log_level

  2. Select Save As > Alert.
  3. In the Save As Alert dialog box, specify the following fields to schedule the alert:

    • Title: Alert Example (Advanced Conditional)
    • Alert Type: Scheduled
      You can also select Real Time for an advanced conditional search.
    • Time Range and Schedule: Select any time range and schedule.
    • Trigger Condition: Custom
    • Custom condition: search count > 10
    Alert advanced conditional schedule.png
  4. Define an action that sends an email that includes the results of the search.
    When you configure a Send Email action that includes search results, the email contains the results of the original base search. It does not include the results of the custom search.

It might appear that you can get the same results if you specify instead, the following search for the base search of a basic conditional search:

log_level=ERROR OR log_level=FATAL OR log_level=CRITICAL) | stats count by log_level | search count > 10

However, a basic conditional alert based on this search provides different results. The search results contain only log level values that are greater than 10. The results from the advanced conditional search details the count for all log levels, but triggers only when the log levels are greater than 10.

Advanced conditional alert for rolling-window alerts

The behavior for advanced conditional alerts differs slightly for a rolling-window alert, which runs in real-time. For a rolling-window alert, the alert triggers when the set condition occurs within the rolling time window of the search.

For the previous example, you can design a rolling-window alert with the same base search and get similar results with the custom condition search. Set the rolling window to a 10 minutes time span. When the real-time search returns 10 log level entries within the 10 minute time span, the alert triggers.

For more examples of scheduled alerts, see "Alert examples," in this manual.

PREVIOUS
Alert types and scenarios
  NEXT
Create per-result alerts

This documentation applies to the following versions of Splunk® Enterprise: 6.3.0, 6.3.1, 6.3.2, 6.3.3, 6.3.4, 6.3.5, 6.3.6, 6.3.7, 6.3.8, 6.3.9, 6.3.10, 6.3.11, 6.3.12, 6.3.13, 6.3.14


Comments

Please ignore my previous comment; I had the proverbial wrong end of the stick. Splunk cron *does* account for a user's timezone settings when running scheduled reports/alerts etc. The search head's timezone does not matter.

Gcato
September 29, 2015

Hi, it would be worthwhile adding some information that cron schedules for searches/alerts run in the time and timezone of the Splunk daemon running on the search head. Therefore, if your local time is different to the search heads timezone, time offsets will need to be applied to the cron schedule to match your local time.

Gcato
September 29, 2015

Was this documentation topic helpful?

Enter your email address, and someone from the documentation team will respond to you:

Please provide your comments here. Ask a question or make a suggestion.

You must be logged into splunk.com in order to post comments. Log in now.

Please try to keep this discussion focused on the content covered in this documentation topic. If you have a more general question about Splunk functionality or are experiencing a difficulty with Splunk, consider posting a question to Splunkbase Answers.

0 out of 1000 Characters