Configure alert trigger conditions
An alert can search for events on a schedule or in real time, but it does not have to trigger every time search results appear. Trigger conditions help you monitor patterns in event data or prioritize certain events.
Alert triggering and alert throttling
Throttling an alert is different from configuring trigger conditions. When you create trigger conditions, search results are evaluated to check if they match the conditions. If results match the trigger conditions, throttling controls whether triggering is suppressed for a period of time. For more information on throttling, see Throttle alerts.
Workflow for trigger configuration
When configuring alert triggering, it is helpful to consider the following questions.
What event pattern is the alert monitoring?
Trigger conditions evaluate the alert's search results for a particular pattern. This pattern combines result fields and their behavior. For example, you can select one of the built-in field count options, such as Number of Hosts, to focus on the
host field. You can then specify the behavior to monitor, such as when that number drops by five. You can also enter a custom triggering condition.
Does the pattern trigger the alert once or for every result?
When the event pattern happens, the alert can trigger just once or one time for each result in the pattern. You can choose an option depending on the notification or other alert action behavior that you want.
Alert types and triggering options
Both alert types offer trigger configuration options for working with the alert search results. Here is a comparison of available triggering options for each type.
|Alert type||Trigger options||Specifying trigger conditions||How matching results trigger the alert|
|Scheduled||Add trigger conditions to evaluate search results.||Built-in result and field count options or a custom triggering condition||Trigger the alert once each time search results match the specified condition or one time for every matching result.|
|Real-time||Per-result||N/A||By default, alert triggers one time for every matching result.|
|Real-time||Trigger conditions that include a rolling time window.||Built-in result and field count options or a custom condition. Also specify a rolling time window or interval.||Trigger the alert once each time search results match the specified condition, or one time for every matching result.|
How searches and trigger conditions work together
Trigger conditions work as a secondary search to evaluate the alert's initial search results. If the secondary search does not return results, the alert does not trigger. When the secondary search does generate results, the alert triggers.
Depending on the alert actions you choose, you can access information about results that trigger the alert. The secondary search for trigger conditions does not determine what information is available for notifications or other alert actions. Result fields and other information come from the initial base search.
Using the alert base search without trigger conditions can limit the information available for notifications. The following example compares using a base search with a custom triggering condition and using a base search without trigger conditions.
This scheduled alert triggers when there are more than ten urgent log_level events. When the alert triggers, it sends an email with the search results.
Use a search with custom trigger condition
The alert uses this search, with Last 7 days selected in the time range picker.
index=_internal (log_level=ERROR OR log_level=FATAL OR log_level=CRITICAL) | stats count by log_level
The following custom triggering condition is added.
search count > 10
In this scenario, the original search results detail the count for all log levels, but the alert triggers only when the log_level counts are greater than ten. This means that all
log_level counts are available to use as part of an alert notification.
Use a search without a trigger condition
The following search looks similar to the previous example. It generates similar alert triggering behavior. However, it creates different results and limits the
log_level information available to notifications or other alert actions.
log_level=ERROR OR log_level=FATAL OR log_level=CRITICAL) | stats count by log_level | search count > 10
In this case, the search results include only
log_level values that are greater than ten. By comparison, using a search with conditional triggering in the previous example means that results include counts for all
log level fields.
Use a search to trigger when there are no alert events
In the case that you want to be notified if no events trigger an alert, you can do this by using the following search or one similar to it:
<your search for events for this data> earliest=0 latest=now | stats count
When you save this search as an alert, set it to trigger if
count < 0.
Create real-time alerts
This documentation applies to the following versions of Splunk Cloud Platform™: 8.1.2103, 8.2.2106, 8.2.2107, 8.2.2105, 8.2.2109, 8.2.2111, 8.2.2112, 8.2.2201 (latest FedRAMP release), 8.2.2202, 8.2.2203