Alerting Manual

Review triggered alerts

You can see records of your recently triggered alerts in the Alert Manager. The Alert Manager displays records of triggered alerts that have List in Triggered Alerts enabled as an alert action in the alert actions of the Save As alert dialog:

Alternatively, if you are creating or updating an alert in Settings > Searches and Reports, you can enable the List in Triggered Alerts alert action to have that alert's triggered alert records appear in the Alert Manager:

Open the Alert Manager by clicking the Alerts link at the upper right-hand corner of the Splunk UI. It opens in a separate window.

Note: The Alert Manager displays records for triggered alerts that are based on existing reports. They will continue to appear even if you disable the alerting aspect of those searches after the alerts were triggered. The Alert Manager will not display records of triggered alerts that are based on deleted reports, however.

For more information about alerts and alert definition, see "About alerts" in this manual.

Setting alert expiration

Triggered alert records are designed to expire (be automatically deleted from the Alert Manager) after a set period of time. You define triggered alert record expiration periods at the individual alert level.

For example, say you have a Firewall breach alert with an Expiration setting of 2 days. If the Firewall breach alert is triggered at 3pm, the related alert record will be deleted from the Alert Manager at 3pm the two days later.

When an alert is first created it has a default expiration time of 24 hours. This means that the triggered alert records for a new alert will disappear a day after it appears in the Alert Manager unless you give it a different expiration period.

To change the default expiration period for an alert's triggered alert records, go to the detail page for the base report in Manager > Searches and Reports and set Expiration to your desired number of seconds, minutes, hours, or days (or just select one of the pre-defined time ranges from the list).

Using the Alert Manager

You can filter the Alert Manager listing by app, alert severity, and alert type. You can also search for specific keywords using the search box. The keyword search applies to fired alert names (which are the same as the names of the searches or reports upon which the alerts are based) and the alert severity (so you can search specifically for alerts of Critical severity, if necessary).

Additionally, the Alert Manager enables you to manually delete individual alert records.

Note that the Severity column enables you to quickly spot those alert records that have been given a higher severity level (such as High or Critical). When you define or update your alert definition, use the Severity field to set the alert severity level. The severity label is for informational purposes only; there is no additional functionality associated with it.

The Type column indicates whether the alert is running in Real-time or is Scheduled to run on a regular interval. The Mode column indicates whether the alert represents a set of events (Digest) or a single event (Per Result).

Click View results for a specific alert record to see the results captured by that alert in another browser tab. This is a search job artifact; it won't contain any events that weren't returned by the search job that originally triggered the alert.

Click Edit search for a specific alert record to edit the underlying search for the alert. You can change the search string and/or redefine the alert definition.

Setting up Alert Manager tracking when upgrading from a pre-4.2 Splunk version

If you're upgrading from a 4.1.x version of Splunk, be aware that by default existing alerts do NOT show up in the Alert Manager. To quickly update your existing alerts so that they show up in the Alert Manager, edit the relevant copy of savedsearches.conf. Add alert.track = true to the stanzas of each report that you have set up as an alert and want to see tracked in the Alert Manager. Review "About configuration files" in the Admin Manual for details about configuration files.