ITSI alert and episode monitoring

The Content Pack for ITSI Monitoring and Alerting ships with several pre-built configurations intended to help Operations teams gain real-time, environment-wide visibility and understanding of incoming alerts. This functionality supplements the default views and dashboards within ITSI so that Operations teams can quickly answer the following questions:

• Is the volume of incoming alerts higher, lower, or the same as what I typically see?
• Which hosts, checks, KPIs, and Services are contributing to the highest volumes of alerts and episodes?
• During an alert storm, what types of alerts are major contributors to the sudden increase in alert volume?

Configuration to detect alert and episode storms

Alert and episode storm detection is driven by the following three services, which also provide regular monitoring of incoming alert and episode volumes for long-term trend analysis and greater visibility. You must enable all three services.

Service Name	Service Purpose
ITSI Event Analytics Service	This is the parent service of the other two services and serves as the top-level node of the alert and episode monitoring service tree.
ITSI Alert Analytics Service	This service tracks incoming alerts and changes to critical status when the volume of incoming alerts rises significantly higher than historical baselines. The service also splits incoming alerts by several key fields to help operations teams quickly identify what values may be contributing to the incoming alert volume. An included ITSI Alert Analytics Template supports greater customization of the ITSI Event Analytics service tree.
ITSI Episode Analytics Service	This service tracks newly-created and open episodes. It changes to critical status when the volume of newly-created episodes rises significantly higher than historical baselines, or when the number of open critical episodes rises significantly higher than historical baselines. The service also splits episodes by several key fields to help operations teams quickly identify what values may be contributing to the episode volume. An included ITSI Episode Analytics Template supports greater customization of the ITSI Event Analytics service tree.

In addition to the services and templates specified in the table above, the Content Pack for ITSI Monitoring and Alerting also includes a saved service analyzer called ITSI Alert and Episode Monitoring that provides fast access to a filtered view of ITSI Event Analytics service tree.

Initial setup

To configure the ITSI Alert and Episode Monitoring Services, perform the following steps:

1. The ITSI Episode Analytics service depends on specific entities and entity filtering for proper operation. If you have created and are using custom aggregation policies beyond those included with ITSI and the Content Pack for Monitoring and Alerting, you need to create new entities for each aggregation policy. The content pack includes a search IT Service Intelligence - CPMA ITSI Aggregation Policies - Entity Discovery Search that can be used to help you create these entities. Go to ITSI > Configuration > Entity Management > Create Entity > Import from search.
2. Choose Saved Search for the search type and locate the saved search named IT Service Intelligence - CPMA ITSI Aggregation Policies - Entity Discovery Search.
3. Use the following mapping values when importing. You can schedule the entity discovery search or run it as needed when new aggregation policies are enabled.

   entity_title -> Entity Title
   itsi_policy_title -> Entity Alias
   itsi_policy_id -> Entity Alias

4. Once entities have been created for each aggregation policy in use, you should then enable all three services, ITSI Event Analytics, ITSI Alert Analytics, and ITSI Episode Analytics.

Optionally, you may re-compute adaptive thresholds for the alert and episode storm KPIs after backfill is complete.

Backfilling KPIs that are configured with adaptive thresholds may result in inaccurate severity values until the nightly adaptive threshold job runs or you manually apply adaptive thresholds in the KPI build.

Customize ITSI event analytics services

The following customizations are supported:

• Tune the alert storm thresholds to better meet your organization's needs. Alert and Episode storm KPIs leverage custom threshold templates (EA - Storm Detection (percent) and EA - Storm Detection (stddev) ) which should be tuned to meet your organizations normal alert volume trends.
• Add additional alert and episode KPIs, split by other meaningful fields. It may be helpful to create additional KPIs where incoming alerts and episodes are split by fields relevant to your organization, to provide greater visibility within the service tree during an alert storm.
• Filter alert and episode KPI base searches to reduce noise. You may have certain types of alerts that you want to omit from the alert and episode storm detection logic. Modify the KPI base searches to eliminate alerts you know you want to ignore.
• Extend the service tree using the service templates and service tree creation searches. The default service tree can be further extended to separate episode monitoring per aggregation policy. To extend the default service tree further, go to ITSI -> Configuration -> Services -> Create Service -> Import from search. Choose Saved Search for the search type and locate the saved search named IT Service Intelligence - CPMA ITSI Event Analytics - Service Tree Search.

Use the following mapping values when importing. You can schedule the service tree creation search or run it as needed when new aggregation policies are enabled.

Service Title -> Service Title
Dependent Services -> Dependent Services
Service Template Link -> Service Template Link
entity_info_itsi_policy_id -> itsi_policy_id