Synchronize KPI searches in ITSI

By default, ITSI staggers the search scheduling of KPIs in order to reduce search load. For example, if you have five KPIs that are scheduled to run every 5 minutes, the search to update the value of each KPI from the summary index is staggered over the 5 minute interval (the first KPI at minute 1, the second KPI at minute 2, and so on).

You can synchronize KPI searches so they update at the same time during the scheduled interval. For example, if 5 KPIs are scheduled to run every 15 minutes, they ALL run at the 15/30/45/00 mark instead of being staggered over the interval.

Prerequisites

• Only users with file system access, such as system administrators, can synchronize KPI searches.
• Review the steps in How to edit a configuration file in the Admin Manual.
• If you are using Splunk Cloud Platform, you must file a support ticket in order to complete the steps below.

Never change or copy the configuration files in the default directory. The files in the default directory must remain intact and in their original location.

Steps

1. Open or create a local itsi_settings.conf in $SPLUNK_HOME/etc/apps/SA-ITOA/local/
2. Add the following stanza:

   [synced_kpi_scheduling]
   disabled = 0
   

When disabled = 0, newly created KPI saved searches run at the same time during each scheduled interval. After a KPI is updated, its search schedule will be overwritten to follow the synchronized schedule.

To reset the search schedules of all existing KPIs to use the new synchronized search schedule after you set disabled = 0, restart Splunk software and then use mode 4 of the kvstore_to_json.py script. See Regenerate KPI search schedules (mode 4) for details.

This setting affects all KPIs, including base searches, for all services. Enabling synced scheduling can have a significant performance impact because it increases the scheduler load. The increased load can result in a delay in search execution due to the number of searches being dispatched at the same time. You might need to scale up your hardware in order to support the increased load.

Regenerate KPI search schedules (mode 4)

The kvstore_to_json.py mode 4 option regenerates the search schedules for your KPIs. Use this command if you have set your KPI saved search schedules to run at the same time in itsi_settings.conf. Run this command to reset the search schedules of all your KPIs to use the new search schedule. See Synchronize KPI searches in ITSI for more information.

1. Run kvstore_to_json.py in mode 4.
   For example:

   cd $SPLUNK_HOME
   bin/splunk cmd python3 etc/apps/SA-ITOA/bin/kvstore_to_json.py -m 4
   

2. Enter the requested information at the prompts (default interactive mode only).
3. You'll see the following message after the KPI search schedules have been reset:

   Retrieving KPIs to reset their saved search scheduling
   Saving updated KPI scheduling
   Done.