Splunk® IT Service Intelligence

Administer Splunk IT Service Intelligence

Download manual as PDF

Download topic as PDF

Create multi-KPI alerts in ITSI

A multi-KPI alert is an alert based on trigger conditions that you define for multiple KPIs. When trigger conditions occur simultaneously for each KPI, a correlation search generates a notable event.

Multi-KPI alerts are useful for correlating the status of multiple KPIs across multiple services. This lets you identify causal relationships, investigate root cause, and provide insights into behaviors across your infrastructure.

For example, to avoid the negative impact that a large spike in traffic can have on website performance, you might create a multi-KPI alert based on two common KPIs: CPU load percent and web requests. A sudden simultaneous spike in both CPU load percent and web request KPIs might indicate a DDOS (Distributed Denial of Service) attack. Multi-KPI alerts can bring such trending behaviors to your attention early, so that you can take action to minimize any impact on performance.

Multi-KPI alerts can apply to one or more KPIs across one or more services.

Multi-KPI alert types

Splunk IT Service Intelligence (ITSI) provides two types of multi-KPI alerts:

Alert Description
Status over time Based on the severity level of one or more KPIs. If one or more trigger conditions for each KPI are satisfied simultaneously, ITSI triggers an alert and generates a notable event.
Composite score Based on a composite score of all KPIs added to the alert. Composite KPI scores are calculated from the KPI severity-level status and an importance value that you assign to the KPI when you create the alert.

Create a multi-KPI alert by selecting Multi-KPI Alerts from the top menu bar. You can also create one from a deep dive by selecting one or more KPI swim lanes and then selecting Bulk Actions > Create Multi KPI Alert.

Status over time alerts

Status over time alerts are multi-KPI alerts based on the severity level status of one or more KPIs. Status over time alerts require you to define one or more trigger conditions per selected KPI.

For example, you select two KPIs for your alert. The first KPI has a trigger for critical 15% of the time and a trigger for high 50% of the time. The second KPI has a single trigger for critical 25% of the time. The boolean expression would read:

(kpi1_perc_crit > 15% OR kpi1_perc_high > 50%) AND (kpi2_perc_critical > 25%)

Create a status over time alert

Prerequisites

You must have the write_itsi_correlation_search capability to create multi-KPI alerts.

Steps

  1. On the Multi-KPI Alerts page, click Status over time.
  2. In the Services column, select the service(s) that contain the KPIs for which you want to set up an alert. Note that only services that belong to teams for which you have read access are listed.
  3. Optionally, select Depends on or Impacts to include KPIs from services that impact or depend on that service.
    All KPIs in each selected service appear in the "KPIs in Selected Services" section.
  4. In the "KPIs in Selected Services" section, click +Add to add a KPI to the alert.
  5. Set trigger conditions for the KPI alert. The condition statement dynamically updates based on the the trigger condition settings.
    Triggers.png
  6. Click Apply. The selected KPI appears in the Selected KPIs section.
  7. Add additional KPIs to the alert. Set one or more trigger conditions for each KPI.

    At least one trigger condition for each KPI must be satisfied simultaneously to trigger the alert and generate a notable event.

  8. Click Save. The Create Correlation Search modal opens.
  9. Configure the correlation search parameters to determine the schedule type, how often the search runs, and the severity of the notable event generated by the alert.
  10. Click Save. ITSI saves the correlation search in Settings > Searches, reports, and alerts. For more information, see Create correlation searches in ITSI.

Composite alerts

Composite KPI alerts are based on a composite score of all KPIs added to the alert. Unlike service health scores, which give you a weighted average of all KPIs in a service, composite KPI scores give you the weighted average of the selected subset of KPIs only.

Composite KPI scores are calculated from the KPI severity-level status and an importance value that you assign to the KPI when you create the alert. This lets you create unique alerts for different groups of KPIs based on the relative importance of a KPI in the context of the alert.

The Importance value that you set for a KPI when you create a composite KPI alert does not change the Importance value that you set for the KPI (which is used to calculate the service health score) when you first create the KPI.

Create a composite KPI alert

Prerequisites

You must have the write_itsi_correlation_search capability to create multi-KPI alerts.

Steps

  1. On the Multi-KPI Alerts page, click Composite score.
  2. In the Services column, select the service(s) that contain the KPIs for which you want to set up an alert. Note that only services that belong to teams for which you have read access are listed. The KPIs in each service appear in the "KPIs in Selected Services" section.
  3. Click +Add to select KPIs that you want to include in the composite KPI alert. The selected KPIs appear in the "Selected KPIs" section.
  4. Set an Importance factor for each KPI. The composite score for all selected KPIs dynamically updates based on the selected Importance factors. CompMKPI.png
  5. Click Save. The Create Correlation Search dialog appears.
  6. Configure correlation search parameters:
    Parameter Description
    Alert me if Defines the suppression conditions for the alert.
      • Score - If the composite score value is less than the value you specify, the alert triggers.
      • Status - If the composite score severity level meets the condition you select, the alert triggers.
    Suppression Ignores notable events generated for the same alert condition. Enable suppression to minimize the number of duplicate notable events sent to Episode Review.
    Trigger if Generate notable events based on trigger conditions that occur simultaneously.
      • Count - If the number of occurrences of the alert condition meets or exceeds the value over time that you specify, the alert triggers.
      • Consecutive Count - If the number of consecutive occurrences of the alert condition meets or exceeds the value that you specify, the alert triggers.
  7. Click Next and specify how the alert should appear in Episode Review.
  8. Click Save.

ITSI saves the correlation search in Configure > Correlation Searches and Settings > searches, reports, and alerts. For more information, see Create correlation searches in ITSI in this manual.

For an end-to-end troubleshooting scenario that involves creating a multi-KPI alert, see Troubleshoot an outage in ITSI in the Splunk IT Service Intelligence Use Cases manual.

PREVIOUS
Configure correlation searches in ITSI
  NEXT
Schedule maintenance downtime in ITSI

This documentation applies to the following versions of Splunk® IT Service Intelligence: 4.0.0, 4.0.1, 4.0.2, 4.0.3, 4.0.4, 4.1.0, 4.1.1, 4.1.2, 4.1.5, 4.2.0, 4.2.1, 4.2.2, 4.2.3, 4.3.0, 4.3.1


Was this documentation topic helpful?

Enter your email address, and someone from the documentation team will respond to you:

Please provide your comments here. Ask a question or make a suggestion.

You must be logged into splunk.com in order to post comments. Log in now.

Please try to keep this discussion focused on the content covered in this documentation topic. If you have a more general question about Splunk functionality or are experiencing a difficulty with Splunk, consider posting a question to Splunkbase Answers.

0 out of 1000 Characters