Splunk® Enterprise

Alerting Manual

Download manual as PDF

This documentation does not apply to the most recent version of Splunk. Click here for the latest version.
Download topic as PDF

About alerts

An alert is an action that a saved search triggers based on the results of the search. When creating an alert, you specify a condition that triggers the alert. Typically the action is an email based on the results of the search. But you can also choose to run a script or to list the alert as a triggered alert in Settings. When you create an alert you are creating a saved search with trigger conditions for the alert.

To avoid sending out alerts too frequently, specify a throttle condition for an alert.

The following list describes the types of alerts:

  • Per result alert. Based on a real-time search. The trigger condition is whenever the search returns a result.
  • Scheduled alert. Runs a search according to a schedule that you specify when creating the alert. You specify results of the search that trigger the alert.
  • Rolling-window alert. Based on a real-time search. The trigger condition is a combination of specified results of the search within a specified time window.

Alert scenarios

This section discusses the scenarios for each type of alert.

Per result alert

Use a per result alert to notify when a real-time search returns a result that matches a condition. Typically, you specify a throttle condition so that the alert triggers only once for a specified time period. Per result examples include the following:

  • Trigger an alert for every failed login attempt.
  • Trigger an alert when a specific type of error occurs on any host.
    You can choose field values that suppress hosts for which you do not want an alert notification.
  • Trigger an alert when a CPU on a host sustains 100% utilization for an extended period of time.
Caution: Be careful using a per result alert in a high availability deployment. If a peer is not available, a real-time search does not warn that the search might be incomplete. Use a scheduled alert for this scenario.

Scheduled alert

Use a scheduled alert to notify when a scheduled search returns results that meet a specific condition. A scheduled alert is useful when an immediate reaction to the alert is not a priority. Scheduled alert examples include:

  • Trigger an alert that runs daily, notifying when the number of items sold that day is less than 500.
  • Trigger an alert that runs hourly, notifying when the number of 404 errors in any hour exceeds 100.

Rolling-window alert

Use a rolling window alert to monitor the results of a real-time search within a specified time interval. For example, monitor the results every 10 minutes or every four hours. Rolling-window alert examples include:

  • Trigger an alert when a user has three consecutive failed logins within a 10 minute period.
    You can set a throttle condition to suppress an alert to once an hour from any user.
  • Trigger an alert when a host is unable to complete an hourly file transfer to another host.
    Set a throttle condition so the alert fires only once every hour for any specific host.
Caution: Be careful using a real-time search in a high availability deployment. If a peer is not available, a real-time search does not warn that the search might be incomplete. Use a scheduled alert for this scenario.

Use throttling to limit alerts

Use throttling to reduce the frequency at which an alert triggers. An alert can trigger frequently based on similar results that the search returns. The schedule to run an alert can also cause the alert to trigger frequently. To reduce the frequency of the alert firing, configure the following:

  • A time period in which to suppress results.
  • Field values that the search returns.

For example, you can create an alert that fires when a system error occurs. For this example, assume that when the system error occurs, it occurs 20 or more times each minute. However, you want to send an alert only once every hour. To reduce the frequency of the alert firing, configure throttling for the alert.

  1. From the Search Page, enter the following search:

    index=_internal log_level=ERROR

  2. Select Save As > Alert
  3. For Result Type, click Real Time to configure a per-result alert.
  4. Click Next.
  5. Select the actions you want to enable.
  6. Select Throttle.
  7. Enter log_level to suppress the alert for the field log_level.
    You can configure throttling to suppress on more than one field. Use a comma-delimited list to specify fields for throttling.
  8. Enter 1 hour as the time to suppress triggering for the alert.

    Alert throttle.png
  9. Click Save.

You can set up a per-result alert that throttles events that share the same clientip and host values. For example, a real-time search with a 60 second window triggers an alert every time an event with disk error appears. Ten events with the error message that occurs in the window triggers five disk error alerts, which is ten alerts within one minute. If the alert sends an email notification each time it triggers, you can overwhelm a email Inbox.

You can configure throttling so that when one alert of this type triggers, it suppresses all successive alerts of the same type for the next 10 minutes. After each successive 10 minutes period pass, the alert can trigger again.

Throttling scheduled and real-time searches

For scheduled searches that run on a frequent basis, and you do not want to be notified for each run, set the throttling controls to suppress the alert to a longer time window.

For real-time searches, if you configure an alert so that it fires once for a trigger condition, you do not need to configure throttling. If the alert fires more than once for the trigger condition, consider throttling to suppress results.

When you configure throttling for a real-time search, start with a throttling period that matches the length of the base search's time window. Expand the throttling period if necessary. This prevents multiple notifications for a given event.

Scheduled reports compared to alerts

A scheduled report is similar to an alert. A scheduled report can execute an action each time the scheduled report runs. For example, a scheduled report that sends a daily report of failed log-ins sends an email even if there are no failed log-ins.

An alert executes its action only when it meets specified conditions. An alert to notify of failed log-ins each hour does not send an email if there are no failed log-ins for a specific hour.

For more information, see Schedule reports, in the Reporting Manual.

Permissions to create and share alerts

The ability to run an alert and edit an alert depend on the user role and capabilties assigned to that role. For information about managing permissions for knowledge objects such as alerts, see Manage knowledge object permissions in the Knowledge Manager.

By default, only users with the Admin or Power roles can do the following:

  • Create alerts.
  • Run real-time searches.
  • Schedule searches.
  • Save searches.
  • Share alerts.

You can share an alert in an app so other users of the app can take advantage of the alert. You share an alert by editing the permissions of the alert.

When you share an alert for a user who is not an Admin user or Power user, make sure the user has the necessary permissions to access the alerting features. For example, make sure the user has the capability to run a real-time search if you share a per result alert.

When you create an alert, you can edit permissions for the alert after you configure it. You can later edit permissions for the alert from the Alerts page. You can share an alert from the Alerts page:

  1. Go to the Alerts page.
  2. For the alert you want to share, select Edit > Edit Permissions.

    Alert permissions.png
  3. Select for whom to display the alert.

    • Owner. This choice makes the alert private to the creator of the alert.
    • App. Display the alert for all users of the app.
    • All Apps. Display the alert for all users of the Splunk Enterprise instance.
  4. Select read and write permissions for the user roles listed in the Permissions Window.

    • Read: Users can see the alert in the Alerts Page and run the alerts in the app.
    • Write: Users with the appropriate permissions can modify, enable, and disable the alert.
  NEXT
Create per-result alerts

This documentation applies to the following versions of Splunk® Enterprise: 6.2.0, 6.2.1, 6.2.2, 6.2.3, 6.2.4, 6.2.5, 6.2.6, 6.2.7, 6.2.8, 6.2.9, 6.2.10, 6.2.11, 6.2.12, 6.2.13, 6.2.14, 6.2.15


Was this documentation topic helpful?

Enter your email address, and someone from the documentation team will respond to you:

Please provide your comments here. Ask a question or make a suggestion.

You must be logged into splunk.com in order to post comments. Log in now.

Please try to keep this discussion focused on the content covered in this documentation topic. If you have a more general question about Splunk functionality or are experiencing a difficulty with Splunk, consider posting a question to Splunkbase Answers.

0 out of 1000 Characters