Alert examples

Use these examples to learn how to use alert types and triggering options. Each example includes a summary of the alerting use case and components. The examples also include steps for creating the alerts.

---

Scheduled alert example

A scheduled alert searches for events on a regular basis. It triggers an alert action if results meet the conditions that you specify.

---

Alert example summary

Use case

Track errors on a Splunk instance. Send an email notification if there are more than five errors in a twenty-four hour period.

Alert type

Scheduled

Search

Look for error events in the last twenty-four hours.

Schedule

Run the search every day at the same time. In this case, the search runs at 10:00 A.M.

Trigger conditions

Trigger the alert action if the search has more than five results.

Alert action

Send an email notification with search result details.

---

Set up the alert

1. From the Search Page, create the following search.

   index=_internal " error " NOT debug source=*splunkd.log* earliest=-24h latest=now
   

2. Select Save As > Alert.
3. Specify the following values for the fields in the Save As Alert dialog box.
   • Title: Errors in the last 24 hours
   • Alert type: Scheduled
   • Time Range: Run every day
   • Schedule: At 10:00
   • Trigger condition: Number of Results
   • Trigger when number of results: is greater than 5.
4. Select the Send Email alert action.
5. Set the following email settings, using tokens in the Subject and Message fields.
   • To: email recipient
   • Priority: Normal
   • Subject: Too many errors alert: $name$
   • Message: There were $job.resultCount$ errors reported on $trigger_date$.
   • Include: Link to Alert and Link to Results
     
     Accept defaults for all other options.
6. Click Save.

---

Real-time alert example

A real-time alert searches continuously for results in real time. You can configure real-time alerts to trigger every time there is a result or if results match the trigger conditions within a particular time window.

---

Alert example summary

Use case

Monitor for errors as they occur on a Splunk instance. Send an email notification if more than five errors occur within one minute.

Alert type

Real-time

Search

Look continuously for errors on the instance.

Trigger conditions

Trigger the alert if there are more than five search results in one minute.

Alert action

Send an email notification.

---

Set up the alert

1. From the Search Page, create the following search.

   index=_internal " error " NOT debug source=*splunkd.log*

2. Select Save As > Alert.
3. Specify the following values for the alert fields.
   • Title: Errors reported (Real-time)
   • Alert type: Real-time
   • Trigger condition: Number of Results
   • Trigger if number of results: is greater than 5 in 1 minute.
4. Select the Send email alert action.
5. Specify the following email settings, using tokens in the Subject and Message fields.
   • To: email recipient
   • Priority: Normal
   • Subject: Real-time Alert: $name$
   • Message: There were $job.resultCount$ errors.
   • Include: Link to Alert, Link to Results, Trigger Condition, and Trigger Time.
   Accept defaults for all other options.
6. Click Save.

---

Throttle the real-time alert

Throttle an alert to reduce its triggering frequency and limit alert action behavior. For example, you can throttle an alert that generates more email notifications than you need.

Throttle the example real-time alert. The following settings change the alert triggering behavior so that email notifications only occur once every ten minutes.

1. From the Alerts page in the Search and Reporting app, select the alert. The alert details page opens.
2. Next to the alert Trigger conditions, select Edit.
3. Select the Throttle option. Specify a 10 minute period.
4. Click Save.

---

Custom trigger condition example

When you create an alert you can use one of the available result or field count trigger condition options. You can also specify a custom trigger condition. The custom condition works as a secondary search on the initial results set.

Alert example summary

Use case

Use the Triggered Alerts list to record WARNING error instances.

Alert type

Real-time

Search

Look for all errors in real-time.

Triggering condition

Check the alert search results for errors of type WARNING. Trigger the alert action if results include any WARNING errors.

Alert action

List the alert in the Triggered Alerts page.

---

Set up the alert

1. From the Search and Reporting home page, create the following search.

   index=_internal source="*splunkd.log" ( log_level=ERROR OR log_level=WARN* OR 
   log_level=FATAL OR log_level=CRITICAL)

2. Select Save As > Alert.
3. Specify the following alert field values.
   • Title: Warning Errors
   • Alert type: Real-time
   • Trigger condition: Custom
   • Custom Condition: search log_level=WARN* in 1 minute
4. Select the List in Triggered Alerts alert action.
5. Click Save.