Define KPI unit and monitoring lag in ITSI
In this step of the KPI setup workflow, define an optional unit of measurement to display for the KPI within glass table visualizations and other dashboards in IT Service Intelligence (ITSI). Configure the monitoring lag to offset indexing lag and improve performance. For an overview of the entire KPI creation workflow, see Overview of creating KPIs in ITSI.
Define the unit of measurement to display in KPI visualizations within service analyzers, deep dive lanes, glass tables, and other dashboards in ITSI populated by the summary index. For example, depending on the statistic you're calculating, you could use GB, Mbps, secs, %, and so on. This setting is optional.
The monitoring lag time, in seconds, is used to offset the indexing lag. Monitoring lag is an estimate of the number of seconds it takes for new events to move from the source to the index. When indexing large quantities of data, an indexing lag can occur, which can cause performance issues. Delay the search time window to ensure that events are actually in the index before running the search.
If you're working with a new data source, click Determine Recommended Lag to sample a 60-minute time period and find out what the minimum, maximum, and recommended monitoring lag setting for your data source is. As a best practice, don't set the monitoring lag to less than 30 seconds.
If the recommended monitoring lag is greater than the KPI frequency, it means there's a difference between the the
_time of the event and the
_indextime when it was written to the indexing tier. For example, you might get a recommended monitoring lag of 350 seconds while the KPI runs every 5 minutes, or 300 seconds. If this difference is large, KPI calculations might be off because the underlying data for that time period might not have been indexed yet. It's best to investigate the cause of the indexing lag and remediate it if possible before proceeding with one of the options below to mitigate issues associated with a high recommended monitoring lag.
Ways to Mitigate High Recommended Monitoring Lag
Perform the following steps to mitigate a high recommended monitoring lag:
- Keep the monitoring lag at 30 seconds and increase the Calculation Window to something greater than the monitoring lag. For example, 5 minutes, 15 minutes, or 24 hours. For explanations of each monitoring calculation, see Configure KPI monitoring calculations in ITSI.
- If the first option doesn't provide the calculation window granularity you need, keep the monitoring lag at 30 seconds and update the KPI search to specify
earliest=-10mto override the Calculation Window setting with the number of minutes you want to look back relative to the current time.
Keep in mind that when there's a difference between the KPI Search Schedule and the Calculation Window, the value of the resulting KPI calculation might be misleading. Therefore, whenever possible:
- Set your KPI Search Schedule and Calculation Window to the same value.
- Use data sources for your KPIs that have a reasonable recommended monitoring lag that's less than the frequency of your KPI.
After you define unit and monitoring lag, move on to step 5: Enable backfill for a KPI in ITSI.
Configure KPI monitoring calculations in ITSI
Enable backfill for a KPI in ITSI
This documentation applies to the following versions of Splunk® IT Service Intelligence: 4.5.0 Cloud only, 4.5.1 Cloud only, 4.6.0 Cloud only, 4.6.1 Cloud only, 4.6.2 Cloud only, 4.7.0, 4.7.1, 4.7.2, 4.8.0 Cloud only, 4.8.1 Cloud only