Splunk® IT Service Intelligence

Service Insights Manual

This documentation does not apply to the most recent version of Splunk® IT Service Intelligence. For documentation on the most recent version, go to the latest release.

Overview of creating KPIs in ITSI

A KPI (Key Performance Indicator) is a recurring saved search that returns the value of an IT performance metric, such as CPU load percentage, memory used percentage, response time, and so on. For an explanation of how KPIs fit into the IT Service Intelligence (ITSI) Service Insights workflow, see Overview of Service Insights in ITSI.

When you create a KPI, you add it directly to a specific service. You can then use KPI search result values inside ITSI to monitor service health, check the status of IT components, and troubleshoot trends that might indicate an issue with your IT systems.

For example, cpu_load_percent is a KPI that measures the CPU load percentage on a server. If your organization has a site uptime guarantee of 99.9% per month, you will need to monitor the status of this KPI and others to ensure that CPU performance remains within acceptable parameters.

Recommended number of KPIs per service

It's not good to have so many KPIs in a single service that you can barely keep track of them. To effectively monitor and troubleshoot a service with 50 or more KPIs, spend time crafting and fostering the KPIs you care about and want to measure, which saves time troubleshooting later.

It's best to have 20 or fewer KPIs per individual service, which is more than enough to capture the key metrics you care about like CPU, IO, disk free, and response time.

Create a KPI

  1. From the ITSI main menu, click Configuration > Services.
  2. Select an existing service.
  3. Go to the KPIs tab.
  4. Click New and choose one of the following options:
    • Select Generic KPI to create a KPI from scratch.
    • Select a KPI template to populate the KPI with a preconfigured source search. KPI templates are tailored for specific service monitoring use cases, such as operating systems, databases, web servers, load balancers. virtual machines, and so on.
  5. Provide a title and description of the KPI. You can't use double quotes ("), or a backslash (/) at the end of the title.

    Note: Unsupported characters in KPI field values include using a dollar sign ($) as the first character, double quotes (") used once, comma (,), equal sign (=), asterisk (*), backtick (`), pipe (|), opening square bracket ([), closing square bracket (]), opening and closing circular brackets. If a comma (,) is used in a field value, each comma-separated value will be treated as a separate value.

KPI scheduled searches with owner: nobody could run based on your server's current time zone to calculate KPI values. To avoid discrepancies with KPI values, check that your source search defines your preferred time zone (for example: EST).

Configure the KPI

To configure a KPI, perform the following high-level steps:

Step Task Description Optional/Required
1 Define a KPI source search A search string that you define as the basis for your KPI, using a data model, an ad hoc search, a metrics search, or a base search. Required
2 Split and filter by entities Break down the KPI to apply the search to multiple entities, enabling comparative analysis of search results on a per-entity basis. Filter entities in or out of the KPI search. Optional
3 Configure KPI monitoring calculations The recurring KPI search schedule and the statistical operations performed on the search results, including service health score calculations. Required
4 Define KPI unit and monitoring lag Define the unit of measurement to display for the KPI. Configure the monitoring lag to offset indexing lag. Optional
5 Enable backfill Fills the summary index with historical raw service health score data. Optional
6 Configure KPI thresholds Severity-level thresholds that you apply to KPI search results. Thresholds let you monitor KPI status (normal, low, medium, high, and critical) and set trigger conditions for alerts. Required
Last modified on 05 June, 2024
Use the Service Analyzer tree view in ITSI   Define a KPI source search in ITSI

This documentation applies to the following versions of Splunk® IT Service Intelligence: 4.11.0, 4.11.1, 4.11.2, 4.11.3, 4.11.4, 4.11.5, 4.11.6, 4.12.0 Cloud only, 4.12.1 Cloud only, 4.12.2 Cloud only, 4.13.0, 4.13.1, 4.13.2, 4.13.3, 4.14.0 Cloud only, 4.14.1 Cloud only, 4.14.2 Cloud only, 4.15.0, 4.15.1, 4.15.2, 4.15.3, 4.16.0 Cloud only, 4.17.0, 4.17.1, 4.18.0, 4.18.1

Was this topic useful?

You must be logged into splunk.com in order to post comments. Log in now.

Please try to keep this discussion focused on the content covered in this documentation topic. If you have a more general question about Splunk functionality or are experiencing a difficulty with Splunk, consider posting a question to Splunkbase Answers.

0 out of 1000 Characters