Splunk® IT Service Intelligence

Service Insights Manual

Acrobat logo Download manual as PDF


Splunk IT Service Intelligence (ITSI) version 4.9.x will reach its End of Life on April 21, 2023. See the Splunk Software Support Policy for details. For information about upgrading to a supported version, see Before you upgrade IT Service Intelligence.
Acrobat logo Download topic as PDF

Configure KPI thresholds in ITSI

Severity-level thresholds determine the current status of your KPI in IT Service Intelligence (ITSI). When KPI values meet or exceed threshold conditions, the KPI status changes, for example from high to critical. The current status of the KPI is reflected in all views across the product, including service analyzers, glass tables, and deep dives. ITSI supports two types of KPI severity-level thresholds: aggregate thresholds and per-entity thresholds.

For an overview of the entire KPI creation workflow, see Overview of creating KPIs in ITSI.

Set aggregate thresholds

Aggregate thresholds are useful for monitoring the status of aggregated KPI values. For example, you might apply aggregate thresholds to monitor the status of KPIs that return the total number of service requests or service errors, based on a calculation that uses the stats count function.

  1. Within the KPI creation workflow, click Aggregate Thresholds.
  2. Click Add threshold to add a range of severity-level thresholds to the threshold preview graph.
    AggThresholds.png
  3. Click Finish.

For information about how KPI importance values affect the overall service health score, see Set KPI importance values in ITSI.

Set per-entity thresholds

Per-entity thresholds are useful for monitoring multiple separate entities against which a single KPI is running. For example, you might have a KPI such as Free Memory % that's running against three separate servers. Using per-entity thresholds, you can monitor the status of Free Memory % on each individual server.

Note: To configure per-entity thresholds, the KPI must be split by entity. For more information, see Split and filter a KPI by entities in ITSI.

  1. Within the KPI creation workflow, click Per-Entity Thresholds.
  2. Click Add threshold and add a range of severity-level thresholds to the preview graph. The preview shows separate search results for each entity associated with the service.
  3. Adjust the thresholds to reflect the severity levels to display when the entities exceed certain limits.
    PerEntityThreshold.png
  4. Click Finish.

Advanced thresholding options

Rather than manually configuring threshold values, you can use one of the following advanced options:

  • Time-based thresholds - user-defined threshold values to be used at different times of the day or week to account for changing KPI workloads.
  • Adaptive thresholds - thresholds calculated by machine learning algorithms that dynamically adapt and change based on the KPI's observed behavior.

For more information, see Overview of advanced thresholding in ITSI.

Apply machine learning-assisted KPI thresholding recommendations

Instead of manually configuring threshold levels or selecting a threshold template that doesn't fit historic KPI data, you can receive threshold recommendations tailored to your KPI data and powered by machine learning. Select the Use Recommended Thresholding Configuration option to receive specific recommendations for which time-based policy and adaptive thresholding algorithm to apply to your KPIs.

The recommended policy will have adaptive thresholding turned on by default, which automatically re-evaluates and updates threshold values as the KPI data changes over time.

Prerequisites

  • Install Python for Scientific Computing in order to use this feature.
  • Your KPI needs at least 30 days worth of backfilled data or display a historical pattern or trend in order to produce recommendations.

Steps

  1. Expand the Thresholding panel for the KPI.
  2. Select Use Recommended Thresholding Configuration.
  3. Configure the following options to load the threshold recommendations:
    Option Description
    Thresholding Direction The direction which the recommended threshold values should increase.
    • Threshold KPI both above and below a certain level: when one of the recommended KPI values go higher or lower than a specified level, the KPI will have a Critical severity.
    • Threshold KPI above a certain level: when one of the recommended KPI values go higher than a specified level, the KPI will have a Critical severity.
    • Threshold KPI below a certain level: when one of the recommended KPI values go lower than a specified level, the KPI will have a Critical severity.
    Analysis Window The time period over which previous KPI data will be analyzed. Recommended threshold values and time policy will be based on the data available in this window.
  4. Select Load Recommendations. Selecting this will overwrite existing threshold settings.
  5. A KPI Recommendations Analysis displays the confidence value of the recommendation. A recommendation with a confidence value between 0.6 to 1.0 is highly recommended.
  6. Select Enable Time Policies to turn on the recommended time policies under the Configure Thresholds for Time Policies panel. The threshold values are updated based on the trends in your KPI data. For example, if your KPI behavior changes on Saturdays and Sundays between 5am and 1pm, the threshold values are updated to account for that behavior.
  7. Select Enable Adaptive Thresholding to compute the standard deviation that will be used for the adaptive thresholding configuration. Select Preview Adaptive Thresholds to see the computed thresholds that displays the calculated threshold pattern against your existing data.
  8. Select Save to save your changes. You can also save the threshold recommendations in your service as part of a service template.

Next steps

  • After you configure KPI thresholds, you can set up alerts to notify you when aggregate KPI severities change. ITSI generates notable events in Episode Review based on the alerting rules you configure. For information, see Receive alerts when KPI severity changes in ITSI.
  • Alternatively, you can set up Anomaly Detection for the KPI. Anomaly Detection uses machine learning algorithms to automatically detect abnormalities in KPI behavior and notify you in Episode Review. For more information, see Apply anomaly detection to a KPI in ITSI.
Last modified on 05 June, 2023
PREVIOUS
Enable backfill for a KPI in ITSI
  NEXT
Set KPI importance values in ITSI

This documentation applies to the following versions of Splunk® IT Service Intelligence: 4.9.0, 4.9.1, 4.9.2, 4.9.3, 4.9.4, 4.9.5, 4.9.6, 4.10.0 Cloud only, 4.10.1 Cloud only, 4.10.2 Cloud only, 4.10.3 Cloud only, 4.10.4 Cloud only, 4.11.0, 4.11.1, 4.11.2, 4.11.3, 4.11.4, 4.11.5, 4.11.6, 4.12.0 Cloud only, 4.12.1 Cloud only, 4.12.2 Cloud only, 4.13.0, 4.13.1, 4.13.2, 4.14.0 Cloud only, 4.14.1 Cloud only, 4.14.2 Cloud only, 4.15.0, 4.15.1, 4.15.2, 4.16.0 Cloud only


Was this documentation topic helpful?


You must be logged into splunk.com in order to post comments. Log in now.

Please try to keep this discussion focused on the content covered in this documentation topic. If you have a more general question about Splunk functionality or are experiencing a difficulty with Splunk, consider posting a question to Splunkbase Answers.

0 out of 1000 Characters