Create KPI base searches in ITSI

KPI base searches let you share a search definition across multiple KPIs in IT Service Intelligence (ITSI). Create base searches to consolidate multiple similar KPIs, reduce search load, and improve search performance. For example, if you have similar ad hoc searches whose only difference is an entity or threshold field, you can consolidate these searches into a single base search definition and achieve better search performance.

ITSI module base searches

ITSI includes several pre-configured KPI base searches based on ITSI modules that you can use with your services. The titles of these base searches begin with "DA-ITSI". KPI base searches that come with ITSI modules are read-only and cannot be modified or deleted. To customize a base search that comes from an ITSI module, clone the base search, then perform your edits on the clone.

Service templates and base searches

Service templates use base searches for their KPIs. When a service template is created from a service, all of the KPIs in the service are imported into the template. Any service KPIs that use ad hoc searches, data model searches, or metrics searches are converted into base searches. These base searches are listed on the KPI Base Searches lister page and are available to use for KPIs in any service, just like any other base search. Base searches that are created for service template KPIs use the following naming standard: <service name>:<KPI name>_<last 8 digits of KPI ID>. For more information about service templates, see Overview of service templates in ITSI.

Create a new base search

You can create new KPI base searches, then use those base searches to build KPIs in the configure services workflow. You can create a base search using an ad hoc search or a metrics search. See Overview of metrics in the Splunk Enterprise Metrics manual for information about Splunk metrics.

Prerequisites

Only users with write permissions to the Global team can create a KPI base search. You must also possess the appropriate capabilities to be able to read, create, or delete base searches.

Steps

1. Click Configuration > KPI Base Searches.
2. Click Create KPI Base Search.
3. Enter a title for your base search. For example, CPU load base search.
4. (Optional) Enter a description for your base search. For example, This base search can be used to build KPIs for CPU metrics.

   You can't change the Team field because all KPI base searches exist in the Global team. All users have read access to the Global team.

5. Click Create.
6. Configure your base search:

   Field	Description
   Search	Select either Ad hoc Search or Metrics Search. For an Ad hoc Search, provide the source ad hoc search. For example, index=os source=ps. For a Metrics Search, select the metrics index to use and the metric. If you do not have a metrics index, you will see the message: "No metrics found".
   KPI Search Schedule	The frequency of the search (Every 1, 5, or 15 minutes).
   Calculation Window	The time period over which the search applies. (Last 1 min, 5 min, 15 min, or a custom value between 1 minute and up to 24 hours).
   Monitoring Lag	Pushes the search back by the specified number of seconds. This value must match the index lag time.
   Split by Entity	Select Yes to maintain a breakdown of KPI values on the entity level.
   Entity Split Field	The field in your search results that is used to look up the corresponding entity to split the KPI. For example, you might want to split your KPI by the processes running on the hosts instead of just the host. The value in this field must be included as part of the Search field in order for filtering to work.
   Filter to Entities in Service	Select Yes to filter the search based on the entity alias. To filter to entities in a service, the service must have associated entities.
   Entity Filter Field	The field in your search results that is used to look up the corresponding entity to filter the KPI. For example, you might want to filter down to all of your database hosts. The value in this field must be included as part of the Search field in order for filtering to work.

7. Click Add Metric. You can add multiple metrics to your base search. Each metric defines a threshold field (for ad hoc searches only) and the calculation method used to aggregate KPI search results on the entity and service level.
8. Configure your metric.

   Field	Description
   Title	The name of the metric. For example, CPU load percent
   Threshold Field (Ad hoc search type only)	This is the field in your data that the KPI aggregates and monitors. For pure counts, use _time.
   Unit	The type of measurement that the KPI calculates. For example, %, MB, and so on.
   Entity Calculation	Sets the calculation method for calculating aggregate search results on the entity level if Split by Entity is set to Yes.
   Service/Aggregate Calculation	Sets the calculation method for calculating the service/aggregate level.
   Fill Data Gaps with	How to treat gaps in your data. This setting affects how aggregate KPI data gaps are displayed in service analyzers, deep dive KPI lanes, glass table KPI widgets, and other dashboards in ITSI populated by the summary index. Select Null values to fill gaps in data with N/A values. Also select the severity level to assign to Null values. Select Last available value to use the last reported value in the ITSI summary index. Aggregate KPI data gaps are filled with the last reported aggregate KPI value. Entity-level data gaps are filled with the corresponding entity's last available value. The search only looks back 30 minutes, so if the gap is larger than 30 minutes it will show N/A. Select Custom value to specify a specific value to use when there is a gap in data. Enter a positive integer. Filled gap values are not used in the calculations performed for Anomaly Detection and Adaptive Thresholding.

9. Click Add to add the metric to the list of metrics defined for your base search. When you build a KPI from a base search, you can select one and only one metric for the KPI.
10. Click Save. You can now use the base search to build KPIs when configuring services.

Build new KPIs from a base search

You can use KPI base searches to build new KPIs. Each KPI that you build is linked to the base search. If you edit and save a base search, those changes are propagated to all linked KPIs.

KPI base searches contain metric specifications. The metric specification for ad hoc base searches includes a threshold field. The metric specification for both ad hoc base searches and metrics base searches contains a method of calculation for aggregate search results at the service and entity level. When you apply a base search to a new KPI, you must select a metric specification from the base search to complete the new KPI search definition.

For example, if you want to create a new KPI to measure CPU load, you might select a metric specification from the base search that contains cpu_load_percent as the threshold field and average as the calculation method.

To build a KPI from a KPI base search:

1. Select Configuration then Services then Service and KPI Management and select a service.
2. In the KPIs tab, select New then Generic KPI.
3. Enter a title and description. Click Next.
4. For KPI Source select Base Search.
5. In the Base Search dropdown, select the base search for your new KPI. You can choose from base search templates provided by ITSI modules, or from your own custom base searches.
6. Select a metric from the Metric menu. Click Next.
   The Entities page appears. All fields are populated from the selected base search. Click Next.
   The Calculation page appears. All fields are populated from the selected base search. Click Next.
   The Optional Setup page appears. All fields are populated from the selected base search, with the exception of Enable Backfill.
7. Select the Enable Backfill check box, then select a Backfill Period (optional). Click Next.
8. Set appropriate severity-level thresholds for the KPI. Click Finish.
   The new KPI is created and appears in the list of KPIs for the service.

   To unlink a KPI from the base search, edit the KPI and change the search type to adhoc, then save the KPI. This lets you use KPI base searches as a starting point for new KPIs.

Delete a KPI base search

When you delete a base search, any service KPIs that use the base search are converted to ad hoc searches. You can't delete a base search that is being used by a KPI in a service template. You must modify those service templates to remove the dependency before you can delete the base search. Additionally, you can't delete a metric that is being used by a base search in a service template.

Prerequisites

• Only users with write permissions to the Global team can delete a KPI base search.
• You must have the delete_itsi_kpi_base_search. The itoa_admin and itoa_team_admin roles have this capability by default. For more information, see Configure users and roles in ITSI.

Steps

1. From the ITSI top menu bar, click Configuration > KPI Base Searches.
2. In the Actions column of the base search you want to delete, click Edit > Delete.
3. Click Delete to confirm.

Wildcards in KPI base searches

As a best practice, don't use wildcards for entities in a KPI base search. When you use wildcards in an entity alias, the entity filter rule returns multiple entities. As a result, those entities act like pseudo entities with N/A entity_keys. This might also lead to an entity broadcasting situation where pseudo entities contribute to the KPI calculations even though that KPI has a strict entity filter. For more information about pseudo entities and entity broadcast, see Manually create pseudo entities in ITSI in the Entity Integrations manual.

As an alternative to using wildcards, consider using entity pivot to associate pseudo entities to a particular service, even through a shared base search without entity broadcast. For examples, see Entity pivot.

KPI base search performance considerations

The performance of KPI base searches is dependent on the following factors:

• The number of KPIs that use the base search.
• The number of services that contain KPIs that use the base search.
• The number of entities matching service entity rules.

Most of the KPI base searches delivered with ITSI are configured to run every minute. Based on testing on a system with 32 cores and 16 GB of memory, a single KPI base search can support up to 5,000 KPIs with 15 entities matched by service entity rules reasonably well.

In general, a KPI base search can support fewer KPIs with many entities or many KPIs with fewer entities. It's not advised to use a single KPI base search for both a high number of KPIs and a high number of entities. As the number of services or matching entities increases, the search runtime also increases.

You can check the runtime for your KPI base searches on the Activity > Jobs page. The runtime is the actual time it takes to run the search. Check the KPI search schedule, or frequency, of the KPI base search. If a search is scheduled to run every minute, and the runtime of that search is longer than 1 minute, the search is taking too long to run.

To reduce a search's runtime, you need to either reduce the number of KPIs using the base search, reduce the number of services that contain the KPI, or reduce the number of entities for each service accordingly. The easiest solution is to clone the KPI base search and use the cloned base search for some of the KPIs.

Fix truncated or incorrect KPI values

Search results are processed, created, and written to the itsi_summary index via an alert action. The default limit on the number of rows that can be written is 50,000 as specified in the $SPLUNK_HOME/etc/system/default/limits.conf file. You can increase this limit if necessary. Calculate the number of the result rows generated by a shared base search using the following formula:

<number of services> x <number of KPIs in each service> x <number of entities per service entity rule> + <number of services> x 2 (one for the service aggregation result, one for the service maximum result)

For example, for 500 services with 10 KPIs in each service and 15 matching entities, the expected number of result rows is 500 x 10 x 15 + 500 x 2 = 76,000 rows

If the number of result rows expected is more than 50,000, ITSI truncates the results and displays incorrect KPI values. If you think you're running into this limitation perform the following steps:

Prerequisites

• Only users with file system access, such as system administrators, can increase write search result limits using a configuration file.
• Review the steps in How to edit a configuration file in the Admin Manual.

Never change or copy the configuration files in the default directory. The files in the default directory must remain intact and in their original location.

Steps

1. Open or create a local limits.conf file at $SPLUNK_HOME/etc/apps/SA-ITOA/local.
2. Add the following stanza:

   [scheduler]
   max_action_results = 1000000
   

3. Set the value for max_action_results to a number higher than 50,000. In the example above it's set to 1,000,000.

Increase the KV store bulk get limit

The KPI base search tries to get all the relevant services from the KV store internally for thresholding related operations. When a KPI base search is attached to a lot of services, the bulk get might reach the KV store bulk get size limit. The default limit is 500MB.

As a guideline, for one service with 20 fully populated KPIs in which all KPIs have custom thresholds with time policies configured, as well as cohesive anomaly detection configured, the size is roughly 0.8 MB in the KV store.

If you have a large number of services containing a lot of KPIs and metadata, it is recommended to increase the KV store bulk get limit in $SPLUNK_HOME/etc/apps/SA-ITOA/local/limits.conf. Increase the max_size_per_result_mb value as necessary.

[kvstore]
# The maximum size, in megabytes (MB), of the result that will be returned for a single query to a collection.
# ITSI requires approximately 50MB per 1,000 KPIs. Override this value if necessary.
# Default: 500 MB
max_size_per_result_mb = 500