Splunk® IT Service Intelligence

Service Insights Manual

Acrobat logo Download manual as PDF


This documentation does not apply to the most recent version of Splunk® IT Service Intelligence. For documentation on the most recent version, go to the latest release.
Acrobat logo Download topic as PDF

Configure KPI monitoring calculations in ITSI

KPI monitoring calculations determine how and when ITSI performs statistical calculations on the KPI. They also determine how ITSI displays gaps in your data. For an overview of the entire KPI creation workflow, see Overview of creating KPIs in ITSI.

Configure the following KPI monitoring calculations:

Field Description
KPI Search Schedule Determines the frequency of the KPI search. Avoid scheduling searches at one minute intervals. Running multiple concurrent KPI searches at short intervals can produce lengthy search queues and is not necessary to monitor most KPIs.
Entity Calculation The method for calculating aggregate search results at the entity level. Each entity has its own alert value based on this calculation type. For example, Average or Maximum. These entity values are then aggregated to create the overall value, which is the value displayed for the KPI.


This setting is only applicable if Split by Entity is set to Yes. For more information, see Split and filter a KPI by entities in ITSI.

Service/Aggregate Calculation The statistical operation that ITSI performs on KPI search results. The correct aggregate calculation to use depends on the type of KPI search. For example, if your search returns results for CPU Load percentage, use Average. if you want a total count of all errors from individual entities, use Sum.
Calculation Window The time period over which the calculation applies. For example, Last 5 Minutes.
Fill Data Gaps with How to treat gaps in your data. This setting affects how KPI data gaps are displayed in service analyzers, deep dive KPI lanes, glass table visualizations, and other dashboards in ITSI populated by the summary index.


For example, if you have a KPI that is split by entity, and one of the entities becomes inactive, the configured value is displayed. However, the configured value does not contribute to the calculation of the KPI alert value.

  • Select Null values to fill gaps in data with N/A values. Also select the severity level to use for Null values.
  • Select Last available value to use the last reported value in the summary index. Aggregate KPI data gaps are filled with the last reported aggregate KPI value. Entity-level data gaps are filled with the corresponding entity's last available value, given that the entity has produced at least one data point. The search looks back up to 30 minutes, so a gap larger than 30 minutes displays as N/A.
  • Select Custom value to indicate a specific value to fill data gaps. The value must be a positive integer. Custom values are currently not used to fill entity-level data gaps.

The values used to fill data gaps are not used in the calculations performed for KPI values, Anomaly Detection, and Adaptive Thresholding.

Next steps

After you define your source search, move on to step 4: Define KPI unit and monitoring lag in ITSI.

Adjust the stateful KPIs caching period

Each time the saved search runs for a KPI with Fill Data Gaps with set to Last available value, ITSI caches the alert value for the KPI in the itsi_kpi_summary_cache KV store collection. A lookup called itsi_kpi_alert_value_cache in the KPI saved search fills entity-level and service-aggregate gaps for the KPI using the cached alert value.

ITSI fills data gaps with the last reported value for at most 30 to 45 minutes, in accordance with the default modular input interval and retention time (15 minutes + 30 minutes). If data gaps for a KPI continue to occur for more than 45 minutes, the data gaps appear as N/A values.

To prevent bloating of the collection with entity and service-aggregate KPI results, a retention policy runs on the itsi_kpi_summary_cache collection using a Splunk modular input. The modular input runs every 15 minutes and removes the entries that have not been updated for more than 30 minutes.

You can change the stateful KPI caching frequency or retention time.

Prerequisites

  • Only users with file system access, such as system administrators, can change the stateful KPI caching frequency and retention time.
  • Review the steps in How to edit a configuration file in the Admin Manual.

Never change or copy the configuration files in the default directory. The files in the default directory must remain intact and in their original location.

Steps

  1. Open or create a local inputs.conf file for the ITSI app at $SPLUNK_HOME/etc/apps/SA-ITOA/local.
  2. Under the [itsi_age_kpi_alert_value_cache://age_kpi_alert_value_cache] stanza, adjust the interval and retentionTimeInSec settings.
Last modified on 18 February, 2022
PREVIOUS
Split and filter a KPI by entities in ITSI
  NEXT
Define KPI unit and monitoring lag in ITSI

This documentation applies to the following versions of Splunk® IT Service Intelligence: 4.11.0, 4.11.1, 4.11.2, 4.11.3, 4.11.4, 4.11.5, 4.11.6, 4.12.0 Cloud only, 4.12.1 Cloud only, 4.12.2 Cloud only, 4.13.1


Was this documentation topic helpful?


You must be logged into splunk.com in order to post comments. Log in now.

Please try to keep this discussion focused on the content covered in this documentation topic. If you have a more general question about Splunk functionality or are experiencing a difficulty with Splunk, consider posting a question to Splunkbase Answers.

0 out of 1000 Characters