Splunk® IT Service Intelligence

Service Insights Manual

Acrobat logo Download manual as PDF


Splunk IT Service Intelligence (ITSI) version 4.12.x reached its End of Life on January 22, 2024. See the Splunk Software Support Policy for details. For information about upgrading to a supported version, see Before you upgrade IT Service Intelligence.
Acrobat logo Download topic as PDF

Define KPI unit and monitoring lag in ITSI

In this step of the KPI setup workflow, define an optional unit of measurement to display for the KPI within glass table visualizations and other dashboards in IT Service Intelligence (ITSI). Configure the monitoring lag to offset indexing lag and improve performance. For an overview of the entire KPI creation workflow, see Overview of creating KPIs in ITSI.

Unit

Define the unit of measurement to display in KPI visualizations within service analyzers, deep dive lanes, glass tables, and other dashboards in ITSI populated by the summary index. For example, depending on the statistic you're calculating, you could use GB, Mbps, secs, %, and so on. This setting is optional.

KPIUnit.png

Monitoring Lag

The monitoring lag time, in seconds, is used to offset the indexing lag. Monitoring lag is an estimate of the number of seconds it takes for new events to move from the source to the index. When indexing large quantities of data, an indexing lag can occur, which can cause performance issues. Delay the search time window to ensure that events are actually in the index before running the search.

Monitoringlag.png



If you're working with a new data source, click Determine Recommended Lag to sample a 60-minute time period and find out what the minimum, maximum, and recommended monitoring lag setting for your data source is. As a best practice, don't set the monitoring lag to less than 30 seconds.

If the recommended monitoring lag is greater than the KPI frequency, it means there's a difference between the the _time of the event and the _indextime when it was written to the indexing tier. For example, you might get a recommended monitoring lag of 350 seconds while the KPI runs every 5 minutes, or 300 seconds. If this difference is large, KPI calculations might be off because the underlying data for that time period might not have been indexed yet. It's best to investigate the cause of the indexing lag and remediate it if possible before proceeding with one of the options below to mitigate issues associated with a high recommended monitoring lag.

Ways to Mitigate High Recommended Monitoring Lag

Perform the following steps to mitigate a high recommended monitoring lag:

  1. Keep the monitoring lag at 30 seconds and increase the Calculation Window to something greater than the monitoring lag. For example, 5 minutes, 15 minutes, or 24 hours. For explanations of each monitoring calculation, see Configure KPI monitoring calculations in ITSI.
  2. If the first option doesn't provide the calculation window granularity you need, keep the monitoring lag at 30 seconds and update the KPI search to specify earliest=-10m to override the Calculation Window setting with the number of minutes you want to look back relative to the current time.

Keep in mind that when there's a difference between the KPI Search Schedule and the Calculation Window, the value of the resulting KPI calculation might be misleading. Therefore, whenever possible:

  • Set your KPI Search Schedule and Calculation Window to the same value.
  • Use data sources for your KPIs that have a reasonable recommended monitoring lag that's less than the frequency of your KPI.

Next steps

After you define unit and monitoring lag, move on to step 5: Enable backfill for a KPI in ITSI.

Last modified on 28 April, 2023
PREVIOUS
Configure KPI monitoring calculations in ITSI
  NEXT
Enable backfill for a KPI in ITSI

This documentation applies to the following versions of Splunk® IT Service Intelligence: 4.11.0, 4.11.1, 4.11.2, 4.11.3, 4.11.4, 4.11.5, 4.11.6, 4.12.0 Cloud only, 4.12.1 Cloud only, 4.12.2 Cloud only, 4.13.0, 4.13.1, 4.13.2, 4.13.3, 4.14.0 Cloud only, 4.14.1 Cloud only, 4.14.2 Cloud only, 4.15.0, 4.15.1, 4.15.2, 4.15.3, 4.16.0 Cloud only, 4.17.0, 4.17.1, 4.18.0, 4.18.1


Was this documentation topic helpful?


You must be logged into splunk.com in order to post comments. Log in now.

Please try to keep this discussion focused on the content covered in this documentation topic. If you have a more general question about Splunk functionality or are experiencing a difficulty with Splunk, consider posting a question to Splunkbase Answers.

0 out of 1000 Characters