Configure KPI monitoring calculations in ITSI
KPI monitoring calculations determine how and when ITSI performs statistical calculations on the KPI. They also determine how ITSI displays gaps in your data. For an overview of the entire KPI creation workflow, see Overview of creating KPIs in ITSI.
Configure the following KPI monitoring calculations:
|KPI Search Schedule
|Determines the frequency of the KPI search. Avoid scheduling searches at one minute intervals. Running multiple concurrent KPI searches at short intervals can produce lengthy search queues and is not necessary to monitor most KPIs.
|The method for calculating aggregate search results at the entity level. Each entity has its own alert value based on this calculation type. For example,
Maximum. These entity values are then aggregated to create the overall value, which is the value displayed for the KPI.
|The statistical operation that ITSI performs on KPI search results. The correct aggregate calculation to use depends on the type of KPI search. For example, if your search returns results for CPU Load percentage, use
Average. if you want a total count of all errors from individual entities, use
|The time period over which the calculation applies. For example,
Last 5 Minutes. You can also set a custom calculation window that ranges from 1 minute to 24 hours.
KPIs with a calculation window greater than 15 minutes can't be backfilled.
|Fill Data Gaps with
|How to treat gaps in your data. This setting affects how KPI data gaps are displayed in service analyzers, deep dive KPI lanes, glass table visualizations, and other dashboards in ITSI populated by the summary index.
The values used to fill data gaps are not used in the calculations performed for KPI values, Anomaly Detection, and Adaptive Thresholding.
Create custom calculation windows
You can specify a custom KPI calculation window (in hours or minutes), when defining your KPI searches in order to fit your specific use case. You can set a window ranging from 1 minute to 24 hours. The maximum window is 24 hours, or 1440 minutes. After you set this custom window, your KPI is evaluated based on the value of the calculation window.
After you define your source search, move on to step 4: Define KPI unit and monitoring lag in ITSI.
Adjust the stateful KPIs caching period
Each time the saved search runs for a KPI with Fill Data Gaps with set to
Last available value, ITSI caches the alert value for the KPI in the itsi_kpi_summary_cache KV store collection. A lookup called itsi_kpi_alert_value_cache in the KPI saved search fills entity-level and service-aggregate gaps for the KPI using the cached alert value.
ITSI fills data gaps with the last reported value for at most 30 to 45 minutes, in accordance with the default modular input interval and retention time (15 minutes + 30 minutes). If data gaps for a KPI continue to occur for more than 45 minutes, the data gaps appear as N/A values.
To prevent bloating of the collection with entity and service-aggregate KPI results, a retention policy runs on the itsi_kpi_summary_cache collection using a Splunk modular input. The modular input runs every 15 minutes and removes the entries that have not been updated for more than 30 minutes.
You can change the stateful KPI caching frequency or retention time.
- Only users with file system access, such as system administrators, can change the stateful KPI caching frequency and retention time.
- Review the steps in How to edit a configuration file in the Admin Manual.
Never change or copy the configuration files in the default directory. The files in the default directory must remain intact and in their original location.
- Open or create a local
inputs.conffile for the ITSI app at
- Under the
[itsi_age_kpi_alert_value_cache://age_kpi_alert_value_cache]stanza, adjust the
Split and filter a KPI by entities in ITSI
Define KPI unit and monitoring lag in ITSI
This documentation applies to the following versions of Splunk® IT Service Intelligence: 4.13.0, 4.13.2, 4.13.3, 4.14.0 Cloud only, 4.14.1 Cloud only, 4.14.2 Cloud only, 4.15.0, 4.15.1, 4.15.2, 4.15.3, 4.16.0 Cloud only, 4.17.0, 4.17.1, 4.18.0