Overview of creating KPIs in ITSI
A KPI (Key Performance Indicator) is a recurring saved search that returns the value of an IT performance metric, such as CPU load percentage, memory used percentage, response time, and so on. For an explanation of how KPIs fit into the IT Service Intelligence (ITSI) Service Insights workflow, see Overview of Service Insights in ITSI.
When you create a KPI, you add it directly to a specific service. You can then use KPI search result values inside ITSI to monitor service health, check the status of IT components, and troubleshoot trends that might indicate an issue with your IT systems.
cpu_load_percent is a KPI that measures the CPU load percentage on a server. If your organization has a site uptime guarantee of 99.9% per month, you will need to monitor the status of this KPI and others to ensure that CPU performance remains within acceptable parameters.
Recommended number of KPIs per service
It's not good to have so many KPIs in a single service that you can barely keep track of them. To effectively monitor and troubleshoot a service with 50 or more KPIs, spend time crafting and fostering the KPIs you care about and want to measure, which saves time troubleshooting later.
It's best to have 20 or fewer KPIs per individual service, which is more than enough to capture the key metrics you care about like CPU, IO, disk free, and response time.
Create a KPI
- From the ITSI main menu, click Configuration > Services.
- Select an existing service.
- Go to the KPIs tab.
- Click New and choose one of the following options:
- Select Generic KPI to create a KPI from scratch.
- Select a KPI template to populate the KPI with a preconfigured source search. KPI templates are tailored for specific service monitoring use cases, such as operating systems, databases, web servers, load balancers. virtual machines, and so on.
- Provide a title and description of the KPI.
KPI scheduled searches with owner: nobody could run based on your server's current time zone to calculate KPI values. To avoid discrepancies with KPI values, check that your source search defines your preferred time zone (for example: EST).
Configure the KPI
To configure a KPI, perform the following high-level steps:
|Define a KPI source search
|A search string that you define as the basis for your KPI, using a data model, an ad hoc search, a metrics search, or a base search.
|Split and filter by entities
|Break down the KPI to apply the search to multiple entities, enabling comparative analysis of search results on a per-entity basis. Filter entities in or out of the KPI search.
|Configure KPI monitoring calculations
|The recurring KPI search schedule and the statistical operations performed on the search results, including service health score calculations.
|Define KPI unit and monitoring lag
|Define the unit of measurement to display for the KPI. Configure the monitoring lag to offset indexing lag.
|Fills the summary index with historical raw service health score data.
|Configure KPI thresholds
|Severity-level thresholds that you apply to KPI search results. Thresholds let you monitor KPI status (normal, low, medium, high, and critical) and set trigger conditions for alerts.
Use the Service Analyzer tree view in ITSI
Define a KPI source search in ITSI
This documentation applies to the following versions of Splunk® IT Service Intelligence: 4.11.0, 4.11.1, 4.11.2, 4.11.3, 4.11.4, 4.11.5, 4.11.6, 4.12.0 Cloud only, 4.12.1 Cloud only, 4.12.2 Cloud only, 4.13.0, 4.13.1, 4.13.2, 4.13.3, 4.14.0 Cloud only, 4.14.1 Cloud only, 4.14.2 Cloud only, 4.15.0, 4.15.1, 4.15.2, 4.15.3, 4.16.0 Cloud only, 4.17.0, 4.17.1, 4.18.0