Configure KPI monitoring calculations in ITSI

KPI monitoring calculations determine how and when ITSI performs statistical calculations on the KPI. They also determine how ITSI displays gaps in your data. For an overview of the entire KPI creation workflow, see Overview of creating KPIs in ITSI.

Configure the following KPI monitoring calculations:

Field	Description
KPI Search Schedule	Determines the frequency of the KPI search. Avoid scheduling searches at one minute intervals. Running multiple concurrent KPI searches at short intervals can produce lengthy search queues and is not necessary to monitor most KPIs.
Entity Calculation	The method for calculating aggregate search results at the entity level. Each entity has its own alert value based on this calculation type. For example, Average or Maximum. These entity values are then aggregated to create the overall value, which is the value displayed for the KPI. This setting is only applicable if Split by Entity is set to Yes. For more information, see Split and filter a KPI by entities in ITSI.
Service/Aggregate Calculation	The statistical operation that ITSI performs on KPI search results. The correct aggregate calculation to use depends on the type of KPI search. For example, if your search returns results for CPU Load percentage, use Average. if you want a total count of all errors from individual entities, use Sum.
Calculation Window	The time period over which the calculation applies. For example, Last 5 Minutes.
Fill Data Gaps with	How to treat gaps in your data. This setting affects how KPI data gaps are displayed in service analyzers, deep dive KPI lanes, glass table visualizations, and other dashboards in ITSI populated by the summary index. For example, if you have a KPI that is split by entity, and one of the entities becomes inactive, the configured value is displayed. However, the configured value does not contribute to the calculation of the KPI alert value. Select Null values to fill gaps in data with N/A values. Also select the severity level to use for Null values. Select Last available value to use the last reported value in the summary index. Aggregate KPI data gaps are filled with the last reported aggregate KPI value. Entity-level data gaps are filled with the corresponding entity's last available value, given that the entity has produced at least one data point. The search looks back up to 30 minutes, so a gap larger than 30 minutes displays as N/A. Select Custom value to indicate a specific value to fill data gaps. The value must be a positive integer. Custom values are currently not used to fill entity-level data gaps. The values used to fill data gaps are not used in the calculations performed for KPI values, Anomaly Detection, and Adaptive Thresholding.

Next steps

After you define your source search, move on to step 4: Define KPI unit and monitoring lag in ITSI.

Adjust the stateful KPIs caching period

Each time the saved search runs for a KPI with Fill Data Gaps with set to Last available value, ITSI caches the alert value for the KPI in the itsi_kpi_summary_cache KV store collection. A lookup called itsi_kpi_alert_value_cache in the KPI saved search fills entity-level and service-aggregate gaps for the KPI using the cached alert value.

ITSI fills data gaps with the last reported value for at most 30 to 45 minutes, in accordance with the default modular input interval and retention time (15 minutes + 30 minutes). If data gaps for a KPI continue to occur for more than 45 minutes, the data gaps appear as N/A values.

To prevent bloating of the collection with entity and service-aggregate KPI results, a retention policy runs on the itsi_kpi_summary_cache collection using a Splunk modular input. The modular input runs every 15 minutes and removes the entries that have not been updated for more than 30 minutes.

You can change the stateful KPI caching frequency or retention time.

Prerequisites

• Only users with file system access, such as system administrators, can change the stateful KPI caching frequency and retention time.
• Review the steps in How to edit a configuration file in the Admin Manual.

Never change or copy the configuration files in the default directory. The files in the default directory must remain intact and in their original location.

Steps

1. Open or create a local inputs.conf file for the ITSI app at $SPLUNK_HOME/etc/apps/SA-ITOA/local.
2. Under the [itsi_age_kpi_alert_value_cache://age_kpi_alert_value_cache] stanza, adjust the interval and retentionTimeInSec settings.