Create multi-KPI alerts in ITSI

A multi-KPI alert in IT Service Intelligence (ITSI) is a type of correlation search that's based on defined trigger conditions for two or more KPIs. When trigger conditions occur simultaneously for each KPI, the search generates a notable event.

For example, to avoid the negative impact that a large spike in traffic can have on website performance, you might create a multi-KPI alert based on two common KPIs: CPU load percent and web requests. A sudden simultaneous spike in both CPU load percent and web request KPIs might indicate a DDOS (Distributed Denial of Service) attack. Multi-KPI alerts can bring such trending behaviors to your attention early, so that you can take action to minimize any impact on performance.

Multi-KPI alerts are useful for correlating the status of multiple KPIs across multiple services. They help you identify causal relationships, investigate root cause, and provide insights into behaviors across your infrastructure.

Multi-KPI alert types

Splunk IT Service Intelligence (ITSI) provides two types of multi-KPI alerts:

Alert	Description
Status over time	Based on the severity level of one or more KPIs. If one or more trigger conditions for each KPI are satisfied simultaneously, ITSI triggers an alert and generates a notable event.
Composite score	Based on a composite score of all KPIs added to the alert. Composite KPI scores are calculated from the KPI severity-level status and an importance value that you assign to the KPI when you create the alert.

Ways to create multi-KPI alerts

There are two ways to create a multi-KPI alert:

• Select Multi-KPI Alerts from the ITSI main menu.
• Within a deep dive, select one or more KPI swim lanes and then click Bulk Actions > Create Multi KPI Alert.

Prerequisites

Your role must have the write_itsi_correlation_search capability to create multi-KPI alerts. The itoa_admin and itoa_team_admin roles have this capability by default.

Create a status over time alert

Status over time alerts are multi-KPI alerts based on the severity level status of one or more KPIs. Status over time alerts require you to define one or more trigger conditions per selected KPI.

For example, you select two KPIs for your alert. The first KPI has a trigger for critical 15% of the time and a trigger for high 50% of the time. The second KPI has a single trigger for critical 25% of the time. The boolean expression would read:

(kpi1_perc_crit > 15% OR kpi1_perc_high > 50%) AND (kpi2_perc_critical > 25%)

1. From the ITSI main menu, click Configuration > Multi-KPI Alerts.
2. Select Status over time.
3. In the Services column, select the service(s) that contain the KPIs for which you want to set up an alert. Note that only services that belong to teams for which you have read access are listed.
4. Optionally, select Depends on or Impacts to include KPIs from services that impact or depend on that service.
   All KPIs in each selected service appear in the "KPIs in Selected Services" section.
5. In the "KPIs in Selected Services" section, click +Add to add a KPI to the alert.
6. Set trigger conditions for the KPI alert. The condition statement dynamically updates based on the the trigger condition settings.
   
7. Click Apply. The selected KPI appears in the Selected KPIs section.
8. Add additional KPIs to the alert. Set one or more trigger conditions for each KPI.

   At least one trigger condition for each KPI must be satisfied simultaneously to trigger the alert and generate a notable event.

9. Click Save. The Create Correlation Search modal opens.
10. Configure the correlation search parameters to determine the schedule type, how often the search runs, and the severity of the notable event generated by the alert.
11. Click Save. ITSI saves the correlation search in Settings > Searches, reports, and alerts. For more information, see Overview of correlation searches in ITSI.

Create a composite alert

Composite KPI alerts are based on a composite score of all KPIs added to the alert. Unlike service health scores, which give you a weighted average of all KPIs in a service, composite KPI scores give you the weighted average of the selected subset of KPIs only.

Composite KPI scores are calculated from the KPI severity-level status and an importance value that you assign to the KPI when you create the alert. This lets you create unique alerts for different groups of KPIs based on the relative importance of a KPI in the context of the alert.

The KPI importance value you set when you create a composite alert doesn't change the original importance value you assigned the KPI when you created it. The original value is still used to calculate service health scores.

1. From the ITSI main menu, click Configuration > Multi-KPI Alerts.
2. Select Composite score.
3. In the Services column, select the service(s) that contain the KPIs for which you want to set up an alert. Note that only services that belong to teams for which you have read access are listed. The KPIs in each service appear in the "KPIs in Selected Services" section.
4. Click +Add to select KPIs that you want to include in the composite KPI alert. The selected KPIs appear in the "Selected KPIs" section.
5. Set an Importance factor for each KPI. The composite score for all selected KPIs dynamically updates based on the selected Importance factors.
   
   
6. Click Save. The Create Correlation Search dialog appears.
7. Configure correlation search parameters:

   Parameter	Description
   Alert me if	Defines the suppression conditions for the alert. Score - If the composite score value is less than the value you specify, the alert triggers. Status - If the composite score severity level meets the condition you select, the alert triggers.
   Suppression	Ignores notable events generated for the same alert condition. Enable suppression to minimize the number of duplicate notable events sent to Episode Review.
   Trigger if	Generate notable events based on trigger conditions that occur simultaneously. Count - If the number of occurrences of the alert condition meets or exceeds the value over time that you specify, the alert triggers. Consecutive Count - If the number of consecutive occurrences of the alert condition meets or exceeds the value that you specify, the alert triggers.

8. Click Next and specify how the alert should appear in Episode Review.
9. Click Save.

ITSI saves the correlation search in Configure > Correlation Searches and Settings > searches, reports, and alerts. For more information, see Overview of correlation searches in ITSI.