Configure KPI thresholds in ITSI
Severity-level thresholds determine the current status of your KPI in IT Service Intelligence (ITSI). When KPI values meet or exceed threshold conditions, the KPI status changes, for example from high to critical. The current status of the KPI is reflected in all views across the product, including service analyzers, glass tables, and deep dives. ITSI supports two types of KPI severity-level thresholds: aggregate thresholds and per-entity thresholds.
For an overview of the entire KPI creation workflow, see Overview of creating KPIs in ITSI.
Set aggregate thresholds
Aggregate thresholds are useful for monitoring the status of aggregated KPI values. For example, you might apply aggregate thresholds to monitor the status of KPIs that return the total number of service requests or service errors, based on a calculation that uses the stats count function.
- Within the KPI creation workflow, click Aggregate Thresholds.
- Click Add threshold to add a range of severity-level thresholds to the threshold preview graph.
- Click Finish.
For information about how KPI importance values affect the overall service health score, see Set KPI importance values in ITSI.
Set per-entity thresholds
Per-entity thresholds are useful for monitoring multiple separate entities against which a single KPI is running. For example, you might have a KPI such as Free Memory % that's running against three separate servers. Using per-entity thresholds, you can monitor the status of Free Memory % on each individual server.
Note: To configure per-entity thresholds, the KPI must be split by entity. For more information, see Split and filter a KPI by entities in ITSI.
- Within the KPI creation workflow, click Per-Entity Thresholds.
- Click Add threshold and add a range of severity-level thresholds to the preview graph. The preview shows separate search results for each entity associated with the service.
- Adjust the thresholds to reflect the severity levels to display when the entities exceed certain limits.
- Click Finish.
Advanced thresholding options
Rather than manually configuring threshold values, you can use one of the following advanced options:
- Time-based thresholds - user-defined threshold values to be used at different times of the day or week to account for changing KPI workloads.
- Adaptive thresholds - thresholds calculated by machine learning algorithms that dynamically adapt and change based on the KPI's observed behavior.
For more information, see Overview of advanced thresholding in ITSI.
Apply machine learning-assisted KPI thresholding recommendations
Instead of manually configuring threshold levels or selecting a threshold template that doesn't fit historic KPI data, you can receive threshold recommendations tailored to your KPI data and powered by machine learning. Select the Use Recommended Thresholding Configuration option to receive specific recommendations for which time-based policy and adaptive thresholding algorithm to apply to your KPIs.
The recommended policy will have adaptive thresholding turned on by default, which automatically re-evaluates and updates threshold values as the KPI data changes over time.
- Install Python for Scientific Computing in order to use this feature.
- Your KPI needs at least 30 days worth of backfilled data or display a historical pattern or trend in order to produce recommendations.
- Select the relevant service from the Services page and edit the service.
- Expand the Thresholding panel for the KPI.
- Configure the following options to load the threshold recommendations:
Option Description Thresholding Direction Sets the base value at which the KPI value is at Critical severity, or at a severity that is not considered normal. Analysis Window The time period over which previous KPI data will be analyzed. Recommended threshold values and time policy will be based on the data available in this window. Select a clean dataset that is the closest to the normal behavior of the KPI, and avoid periods of time where the data contains known abnormal behavior caused by any disruptions, such as outages.
Note: Selecting 7 days of data will help the algorithm detect daily patterns. Selecting 30 days or more (14 days at minimum) helps the algorithm detect weekly patterns in addition to daily patterns.
- Select Load Recommendations. Selecting this will load a preview of the recommended threshold settings which can be tuned further, or saved. Select Preview Adaptive Thresholds to view the calculated threshold pattern against your existing data, using the recommended configuration.
Note: Time policies and adaptive thresholding with a specific training window are automatically turned on when you load recommendations.
- A KPI Recommendations Analysis displays the confidence value of the recommendation. A recommendation with a confidence value between 0.6 to 1.0 is highly recommended.
- (Optional) View the recommended time policies in the Configure Thresholds for Time Policies panel. The threshold values are updated based on the trends in your KPI data. For example, if your KPI behavior changes on Saturdays and Sundays between 5am and 1pm, the threshold values are updated to account for that behavior.
- Select Save to save your changes. You can also save the threshold recommendations in your service as part of a service template to quickly deploy your threshold settings to other KPIs.
- After you configure KPI thresholds, you can set up alerts to notify you when aggregate KPI severities change. ITSI generates notable events in Episode Review based on the alerting rules you configure. For information, see Receive alerts when KPI severity changes in ITSI.
- Alternatively, you can set up Anomaly Detection for the KPI. Anomaly Detection uses machine learning algorithms to automatically detect abnormalities in KPI behavior and notify you in Episode Review. For more information, see Apply anomaly detection to a KPI in ITSI.
Enable backfill for a KPI in ITSI
Set KPI importance values in ITSI
This documentation applies to the following versions of Splunk® IT Service Intelligence: 4.9.0, 4.9.1, 4.9.2, 4.9.3, 4.9.4, 4.9.5, 4.9.6, 4.10.0 Cloud only, 4.10.1 Cloud only, 4.10.2 Cloud only, 4.10.3 Cloud only, 4.10.4 Cloud only, 4.11.0, 4.11.1, 4.11.2, 4.11.3, 4.11.4, 4.11.5, 4.11.6, 4.12.0 Cloud only, 4.12.1 Cloud only, 4.12.2 Cloud only, 4.13.0, 4.13.1, 4.13.2, 4.13.3, 4.14.0 Cloud only, 4.14.1 Cloud only, 4.14.2 Cloud only, 4.15.0, 4.15.1, 4.15.2, 4.15.3, 4.16.0 Cloud only, 4.17.0, 4.17.1