Splunk Cloud Platform

Alerting Manual

Alert examples

Use these examples to learn how to use alert types and triggering options. Each example includes a summary of the alerting use case and components. The examples also include steps for creating the alerts.


Scheduled alert example

A scheduled alert searches for events on a regular basis. It triggers an alert action if results meet the conditions that you specify.


Alert example summary

Use case
Track errors on a Splunk instance. Send an email notification if there are more than five errors in a twenty-four hour period.
Alert type
Scheduled
Search
Look for error events in the last twenty-four hours.
Schedule
Run the search every day at the same time. In this case, the search runs at 10:00 A.M.
Trigger conditions
Trigger the alert action if the search has more than five results.
Alert action
Send an email notification with search result details.

Set up the alert

  1. From the Search Page, create the following search.
    index=_internal " error " NOT debug source=*splunkd.log* earliest=-24h latest=now
    
  2. Select Save As > Alert.
  3. Specify the following values for the fields in the Save As Alert dialog box.
    • Title: Errors in the last 24 hours
    • Alert type: Scheduled
    • Time Range: Run every day
    • Schedule: At 10:00
    • Trigger condition: Number of Results
    • Trigger when number of results: is greater than 5.
  4. Select the Send Email alert action.
  5. Set the following email settings, using tokens in the Subject and Message fields.
    • To: email recipient
    • Priority: Normal
    • Subject: Too many errors alert: $name$
    • Message: There were $job.resultCount$ errors reported on $trigger_date$.
    • Include: Link to Alert and Link to Results

      Accept defaults for all other options.
  6. Click Save.

Real-time alert example

A real-time alert searches continuously for results in real time. You can configure real-time alerts to trigger every time there is a result or if results match the trigger conditions within a particular time window.


Alert example summary

Use case
Monitor for errors as they occur on a Splunk instance. Send an email notification if more than five errors occur within one minute.
Alert type
Real-time
Search
Look continuously for errors on the instance.
Trigger conditions
Trigger the alert if there are more than five search results in one minute.
Alert action
Send an email notification.

Set up the alert

  1. From the Search Page, create the following search.
    index=_internal " error " NOT debug source=*splunkd.log*
  2. Select Save As > Alert.
  3. Specify the following values for the alert fields.
    • Title: Errors reported (Real-time)
    • Alert type: Real-time
    • Trigger condition: Number of Results
    • Trigger if number of results: is greater than 5 in 1 minute.
  4. Select the Send email alert action.
  5. Specify the following email settings, using tokens in the Subject and Message fields.
    • To: email recipient
    • Priority: Normal
    • Subject: Real-time Alert: $name$
    • Message: There were $job.resultCount$ errors.
    • Include: Link to Alert, Link to Results, Trigger Condition, and Trigger Time.
    Accept defaults for all other options.
  6. Click Save.

Throttle the real-time alert

Throttle an alert to reduce its triggering frequency and limit alert action behavior. For example, you can throttle an alert that generates more email notifications than you need.

Throttle the example real-time alert. The following settings change the alert triggering behavior so that email notifications only occur once every ten minutes.

  1. From the Alerts page in the Search and Reporting app, select the alert. The alert details page opens.
  2. Next to the alert Trigger conditions, select Edit.
  3. Select the Throttle option. Specify a 10 minute period.
  4. Click Save.

Custom trigger condition example

When you create an alert you can use one of the available result or field count trigger condition options. You can also specify a custom trigger condition. The custom condition works as a secondary search on the initial results set.

Alert example summary

Use case
Use the Triggered Alerts list to record WARNING error instances.
Alert type
Real-time
Search
Look for all errors in real-time.
Triggering condition
Check the alert search results for errors of type WARNING. Trigger the alert action if results include any WARNING errors.
Alert action
List the alert in the Triggered Alerts page.

Set up the alert

  1. From the Search and Reporting home page, create the following search.
    index=_internal source="*splunkd.log" ( log_level=ERROR OR log_level=WARN* OR 
    log_level=FATAL OR log_level=CRITICAL)
  2. Select Save As > Alert.
  3. Specify the following alert field values.
    • Title: Warning Errors
    • Alert type: Real-time
    • Trigger condition: Custom
    • Custom Condition: search log_level=WARN* in 1 minute
  4. Select the List in Triggered Alerts alert action.
  5. Click Save.
Last modified on 10 June, 2016
Additional alert configuration options   Configure alerts in savedsearches.conf

This documentation applies to the following versions of Splunk Cloud Platform: 8.2.2112, 8.2.2201, 8.2.2202, 8.2.2203, 9.0.2205, 9.0.2208, 9.0.2209, 9.0.2303, 9.0.2305, 9.1.2308, 9.1.2312, 9.2.2403 (latest FedRAMP release), 9.2.2406


Was this topic useful?







You must be logged into splunk.com in order to post comments. Log in now.

Please try to keep this discussion focused on the content covered in this documentation topic. If you have a more general question about Splunk functionality or are experiencing a difficulty with Splunk, consider posting a question to Splunkbase Answers.

0 out of 1000 Characters