Alerting Manual

 


Alert examples

Alert examples

This chapter shows examples of creating various types of alerts.

  • Scheduled alert
  • Real-time alert
  • Custom conditional alert

Scheduled alert

A scheduled alert runs periodically at a scheduled time, responding to a condition that triggers the alert.

This example uses a search to track when there are too many errors in a Splunk Enterprise instance during the last 24 hours. When the number of errors exceeds 5, the alert sends an email with information about the conditions that triggered the alert. The alert sends an email every day at 10:00am when the number of errors exceed the threshold.

  1. From the Search Page, create the following search
    index=_internal " error " NOT debug source=*splunkd.log* earliest=-24h latest=now
  2. Click Save As > Alert.
  3. Specify the following values for the fields in the Save As Alert dialog:

    Title: Errors in the last 24 hours
    Alert type: Scheduled
    Time Range: Run every day
    Schedule: At 10:00
    Trigger condition: Number of Results
    Trigger if number of results: is Greater than 5.

    Alert SchedAlertExample1.png
  4. Click Next.
  5. Click Send Email.
  6. Set the following email settings, using tokens in the Subject and Message fields:

    To: email recipient
    Priority: Normal
    Subject: Too many errors alert: $name$
    Message: There were $job.resultCount$ errors reported on $trigger_date$.
    Include: Link to Alert and Link to Results

    Accept defaults for all other options.

    For more information on tokens, see Use tokens in email notifications

    Alert SchedAlertExample.png
  7. Click Save.
    After you create the alert you can view and edit the alert in the Alerts Page.

When the alert triggers, it sends the following email:

Alert SchedAlertEmail.png

Real-time alert

You can configure a real-time alert to ensure that you get timely updates to the condition that triggers the alert. The procedure to configure a real-time alert is similar to that of a scheduled alert, but contains differences to ensure timely delivery.

In this example, do not specify a time range for the search. The real-time alert specifies when the search runs.

  1. From the Search Page, create the following search:
    index=_internal " error " NOT debug source=*splunkd.log*
  2. Click Save As > Alert.
  3. Specify the following values for the fields in the Save As Alert dialog:

    Title: Errors reported (Real-time)
    Alert type: Real Time
    Trigger condition: Number of Results
    Trigger if number of results: is Greater than 5 in 1 minute.

    Alert RTAlertExample1.png
  4. Click Next.
  5. Click Send Email.
  6. Specify the following email settings, using tokens in the Subject and Message fields:

    To: email recipient
    Priority: Normal
    Subject: Real Time Alert: $name$
    Message: There were $job.resultCount$ errors.
    Include: Link to Alert, Link to Results, Trigger Condition, and Trigger Time.

    Accept defaults for all other options.

    For more information on tokens, see Use tokens in email notifications

    Alert RTAlertExample.png
  7. Click Save.
    After you create the alert you can view and edit the alert in the Alerts Page.


Modify trigger condition

If a search takes longer to run than the time specified in the trigger condition, then the alert could fail to fire. Modify the trigger condition accordingly.

For the previous real-time alert example, assume that the search takes longer than one minute to run. To ensure the alert fires, modify the trigger condition period to 10 minutes.

Modify throttling setting

For some searches, the trigger condition can happen many times during the period configured to fire the alert. For real-time alerts, this can result in numerous emails that can overwhelm your inbox. Use the throttle action to limit the number of emails. For the previous real-time alert example, when configuring alert actions specify a reasonable time to wait before the alert fires. For example:

  1. In the Edit Alert dialog, click Throttle.
  2. For Suppress triggering for enter 10 minutes.

Custom conditional alert

When you create an alert you specify the trigger condition for the alert. The Edit Alert dialog lets you choose from the following trigger conditions.

  • Per result
    Triggers when the search returns a result.
  • Number of results
    Triggers when the search returns a specified number of results.
  • Number of hosts
    Triggers when the search returns a specified number of hosts.
  • Number of sources
    Triggers when the search returns a specified number of hosts.
  • Custom
    Triggers on a custom search condition.

The following example shows how to create an alert with a custom search condition. The example uses a base base search that checks for all errors. The trigger condition is when an error of type WARNING occurs. The alert action lists the triggered alert.

  1. From the Search Page, create the following search
    index=_internal source="*splunkd.log" ( log_level=ERROR OR log_level=WARN* OR log_level=FATAL OR log_level=CRITICAL)
  2. Click Save As > Alert.
  3. Specify the following values for the fields in the Save As Alert dialog:

    Title: Warning Errors
    Alert type: Real-time
    Trigger condition: Custom
    Custom Condition: search log_level=WARN* in 1 minute
  4. Click Next.
  5. Click List in Triggered Alerts.
  6. Click Save.
    After you create the alert you can view and edit the alert in the Alerts Page. When the alert triggers, the Alerts Page lists the alert in the Trigger History section.

This documentation applies to the following versions of Splunk: 6.1 , 6.1.1 , 6.1.2 , 6.1.3 , 6.1.4 , 6.1.5 , 6.2.0 View the Article History for its revisions.


You must be logged into splunk.com in order to post comments. Log in now.

Was this documentation topic helpful?

If you'd like to hear back from us, please provide your email address:

We'd love to hear what you think about this topic or the documentation as a whole. Feedback you enter here will be delivered to the documentation team.

Feedback submitted, thanks!