ITSI alert and episode storm alerting

While the ITSI Event Analytics service tree provides a visual indication of alert and episode health, your Operations team likely wants to be proactively alerted when an alert or episode storm is detected. This section covers how to set up and consume proactive notifications when the system detects alert and episode storms.

See Episodes by ITSI Alert and Episode Monitoring for information about that aggregation policy.

Initial setup

To configure alert storm and episode storm alerting, perform the following steps:

1. Ensure that at least one Service Monitoring correlation search is enabled which will detect degradations in the alert and episode storm KPIs. Splunk recommends either Service Monitoring - KPI Degraded or Service Monitoring - Sustained KPI Degradation; however if you are familiar with the concepts of the content pack, you can configure any correlation search that will detect KPI or Service Health degradations.
2. Ensure the ITSI Alert and Episode Monitoring aggregation policy is enabled.
3. Modify the Episode View Default Tab configuration in the ITSI Alert and Episode Storm Activity Saved Episode Review. Unfortunately, saved episode views in ITSI content packs cannot ship with a specific value in the Episode View Default Tab configuration. Therefore, you must manually update the value from the default value of "Impact" to "Dashboard". This change ensures that when you click on an ITSI Alert and Episode Storm episode, the first tab you see is the Dashboard tab, which contains valuable information about the current alert storm.

ITSI Alert and Episode Storm Activity Saved Episode Review

When alert storms and episode storms occur, the ITSI Event Analytics service tree will degrade, and those degradations will be detected by one or more Service Monitoring correlation searches. Notable events created afterward are grouped together into Episodes by the ITSI Alert and Episode Monitoring aggregation policy. Within the Alerts and Episodes view of ITSI, you can select a pre-built Episode Review called ITSI Alert and Episode Storm Activity which shows episodes that have been created by the ITSI Event Analytics service degradations.

This saved episode view has been specially designed to maximize the value of information presented through the following configurations:

• The saved episode view defaults to showing any episodes created in the last 4 hours for the ITSI Event Analytics service tree.
• The saved episode view contains a customized episode dashboard to display critical alert and episode KPIs across the environment.
• The saved episode view is configured to display a custom dashboard when the episode is opened for viewing so as to help triage the cause of the storm.

Perform triage for alert and episode storms

Once a storm is detected, you use dashboards to triage the storm to determine what might be causing it. The dashboards can be loaded through drilldowns from the custom episode dashboard. Use Click to analyze the incoming alert volume trend for an alert storm and Click to analyze the field values of the alerts for the alert storm.

Customize ITSI alert and episode storm alerting

Functionality described in this section is designed to reduce the need for customizations, but there are additional customizations that you may want to make to suit the needs of your organization.

Similar to the default actions in other aggregation policies shipped with this content pack, the default action for the ITSI Alert and Episode Monitoring aggregation policy is to apply a comment to the episode. You can modify the action to something more meaningful such as sending an email, sending to Splunk On-Call, or creating a ticket in a ticketing system.

The Probable Cause of Storm and Field Values Distribution panels of the ITSI Alert and Episode Storm Episode Dashboard default to reviewing the following fields: src, signature, source, service_name, kpi, entity_title, alert_group. You can configure the content pack to look at other fields beyond these defaults. To modify the fields that are analyzed in these dashboard panels, modify the itsi_get_default_analysis_fields macro to suit your organization's needs by adding new fields or removing any unnecessary default fields.