Splunk Cloud Platform

Alerting Manual

Configure alert trigger conditions

An alert can search for events on a schedule or in real time, but it does not have to trigger every time search results appear. Trigger conditions help you monitor patterns in event data or prioritize certain events.

Alert triggering and alert throttling
Throttling an alert is different from configuring trigger conditions. When you create trigger conditions, search results are evaluated to check if they match the conditions. If results match the trigger conditions, throttling controls whether triggering is suppressed for a period of time. For more information on throttling, see Throttle alerts.


Workflow for trigger configuration

When configuring alert triggering, it is helpful to consider the following questions.

What event pattern is the alert monitoring?
Trigger conditions evaluate the alert's search results for a particular pattern. This pattern combines result fields and their behavior. For example, you can select one of the built-in field count options, such as Number of Hosts, to focus on the host field. You can then specify the behavior to monitor, such as when that number drops by five. You can also enter a custom triggering condition.

Does the pattern trigger the alert once or for every result?
When the event pattern happens, the alert can trigger just once or one time for each result in the pattern. You can choose an option depending on the notification or other alert action behavior that you want.


Alert types and triggering options

Both alert types offer trigger configuration options for working with the alert search results. Here is a comparison of available triggering options for each type.

Alert type Trigger options Specifying trigger conditions How matching results trigger the alert
Scheduled Add trigger conditions to evaluate search results. Built-in result and field count options or a custom triggering condition Trigger the alert once each time search results match the specified condition or one time for every matching result.
Real-time Per-result N/A By default, alert triggers one time for every matching result.
Real-time Trigger conditions that include a rolling time window. Built-in result and field count options or a custom condition. Also specify a rolling time window or interval. Trigger the alert once each time search results match the specified condition, or one time for every matching result.

How searches and trigger conditions work together

Trigger conditions work as a secondary search to evaluate the alert's initial search results. If the secondary search does not return results, the alert does not trigger. When the secondary search does generate results, the alert triggers.

Depending on the alert actions you choose, you can access information about results that trigger the alert. The secondary search for trigger conditions does not determine what information is available for notifications or other alert actions. Result fields and other information come from the initial base search.

Using the alert base search without trigger conditions can limit the information available for notifications. The following example compares using a base search with a custom triggering condition and using a base search without trigger conditions.

Example

This scheduled alert triggers when there are more than ten urgent log_level events. When the alert triggers, it sends an email with the search results.

Use a search with custom trigger condition

The alert uses this search, with Last 7 days selected in the time range picker.

index=_internal (log_level=ERROR OR log_level=FATAL OR log_level=CRITICAL) | stats count by log_level

The following custom triggering condition is added.

search count > 10

In this scenario, the original search results detail the count for all log levels, but the alert triggers only when the log_level counts are greater than ten. This means that all log_level counts are available to use as part of an alert notification.


Use a search without a trigger condition

The following search looks similar to the previous example. It generates similar alert triggering behavior. However, it creates different results and limits the log_level information available to notifications or other alert actions.

log_level=ERROR OR log_level=FATAL OR log_level=CRITICAL) | stats count by log_level | search count > 10

In this case, the search results include only log_level values that are greater than ten. By comparison, using a search with conditional triggering in the previous example means that results include counts for all log level fields.

Use a search to trigger when there are no alert events

In the case that you want to be notified if no events trigger an alert, you can do this by using the following search or one similar to it:

<your search for events for this data> earliest=0 latest=now | stats count

When you save this search as an alert, set it to trigger if count=0 or count < 0.

Last modified on 31 July, 2020
Create Splunk Mobile alerts   Throttle alerts

This documentation applies to the following versions of Splunk Cloud Platform: 8.2.2112, 8.2.2201, 8.2.2202, 8.2.2203, 9.0.2205, 9.0.2208, 9.0.2209, 9.0.2303, 9.0.2305, 9.1.2308, 9.1.2312, 9.2.2403, 9.2.2406 (latest FedRAMP release), 9.3.2408


Was this topic useful?







You must be logged into splunk.com in order to post comments. Log in now.

Please try to keep this discussion focused on the content covered in this documentation topic. If you have a more general question about Splunk functionality or are experiencing a difficulty with Splunk, consider posting a question to Splunkbase Answers.

0 out of 1000 Characters