Create custom threshold windows in ITSI
Create custom KPI threshold windows to account for time periods when your KPI threshold values may vary due to seasonal events, like holidays, in order to avoid false alerts. After creating your custom threshold windows, the new severity levels are applied to linked KPIs once the custom threshold window is saved. The adjustments are also applied when additional KPI configurations, such as those from adaptive thresholding, or links from existing KPI threshold templates or service templates. Any KPIs linked to a custom threshold window display an Active badge indicator on the Service Analyzer and Service configuration pages.
|You can only create a custom threshold window in a team for which you have read and write permissions. Custom threshold windows linked to KPIs associated with ITSI teams can be accessed by all members of that team. For more information about teams, see Overview of teams in ITSI in the Administration Manual.
|Create services and KPIs
|Create services with KPIs to link to your custom threshold windows. For more information, see Overview of creating services in ITSI and Overview of creating KPIs in ITSI.
Follow these steps to create a custom threshold window.
Step 1. Create a custom threshold window
- Select Configuration > Custom Threshold Windows from the ITSI main menu.
- Select Create Custom Threshold Window to create a new window. You can either create a window that runs only once or on a repeating schedule.
- Configure the following fields for your custom threshold window:
Field Description Title The name of the custom threshold window. Each custom threshold window must have a unique name. Description (Optional) Describe your custom threshold window. Schedule How often your custom threshold window runs. Select one of the following options:
- Does not repeat
- Run every week
- Run every month
You can also specify the exact day of the week and hour that you want your window to run.
Window start The date, hour, and minute at which the custom threshold window begins to run. This option only applies to non-recurring custom threshold windows. Window duration The number of days or hours to which the threshold window applies. Select up to 90 days or 2160 hours. Link one KPI to this custom threshold window You can't create a custom threshold window without selecting at least one KPI.
If the Link additional KPIs after creation checkbox is selected, you're redirected to the custom threshold window's configuration page to add additional settings and preview your updated KPI thresholds.
Select at least one service and KPI to link to the custom threshold window. You can't create a custom threshold window without selecting at least one KPI or service. If the 'Link additional KPIs after creation checkbox is selected, you're redirected to the custom threshold window configuration page to add additional KPIs.
- Select Create.
Custom threshold window statuses
Each custom threshold window has one of the following statuses:
- Active: the custom threshold window is running.
- Completed: the custom threshold window completed its schedule run.
- Invalid: the custom threshold window is not linked to any KPIs and therefore not running, or has already completed its scheduled run time. The original KPI(s) it was linked to may have also been deleted. You can delete these custom threshold windows.
- Restoring: the custom threshold window is being restored after a backup.
- Scheduled: the custom threshold window will run at a future time.
- Skipped: something caused the window not to run, or the window time already passed.
- Stopped: the custom threshold window was stopped and no longer running.
Read-only custom threshold windows
Custom threshold window configurations can't be edited under the following conditions:
- When the window status is Completed and the schedule is set to Does not repeat
- When the window status is Invalid or Skipped
Under the following conditions, you can still add or remove linked KPIs, but you cannot make additional edits to the custom threshold window:
- When the window status is Active
- When you do not have permissions to view or edit one or more of the KPIs linked to the custom threshold window
Step 2. Apply threshold configuration
On the configuration page for your custom threshold window, select a value that will increase or decrease the default KPI threshold values by the specified percent value in the Threshold Configuration section. For example, if the range for a Low severity-level KPI threshold is set to 10-100, increasing the range by 10% updates the base range to alert only for values between 11-110.
You can preview the effect of your custom threshold configuration settings against your original KPI threshold values in the Preview section, and view the effect of the custom threshold during a specific time range by selecting an option on the Time Range Window dropdown. Use the preview to determine how much of a threshold adjustment is best for your KPIs.
When a KPI linked to a custom threshold window is updated elsewhere in ITSI, any adjusted thresholds are re-evaluated and changes are automatically applied to the custom threshold window. When you change the values of the base KPI thresholds for KPIs that are already linked to an active custom threshold window, the custom thresholds will be recalculated and applied to the updated base thresholds. For example, if you change a KPI's base threshold values from 10-100 to 20-200, the updated custom thresholds increased by 10% will be 22-220.
Step 3. Link additional KPIs
Select the Linked Services & KPIs button to add additional services and KPIs that you want to link to your custom threshold window. You can only edit a custom threshold window linked to KPIs for which you have read and write permissions.
Note: Permission to edit the custom threshold window is based on the permissions set for the KPIs that you link here.
- Select Link Services & KPIs.
- Select or search for the KPIs that you want to link to this threshold window in the popup modal. Your configuration will apply to these KPIs at the time your window becomes active.
- Select Save.
Note: You can't link KPIs that are already linked to another custom threshold window with the same scheduled time period.
Edit custom threshold windows
If you want to make changes to an existing custom threshold window, you can select one of the following actions on the custom threshold windows lister page:
- Edit: Go to the main configuration page to make changes to the linked KPIs, percentage thresholds, or schedule and duration
- Edit configuration: Edit the schedule and frequency that the window runs
Note: You can still update the linked KPIs but can't update the remaining configuration when a custom threshold window is active.
Stop custom threshold windows
You can stop a custom threshold window that's currently active in cases when you identify a configuration error or want the window to stop adjusting KPI thresholds:
- Select your custom threshold window.
- Change the Threshold Window Status by selecting the Stop button.
- (Optional) Include the reason you want to stop your custom threshold window.
- Select Stop.
The KPI linked to the custom threshold window will no longer display as active and the custom threshold window's status will be updated to Stopped. You will not be able to make edits to a custom threshold window once it is stopped.
Backup and Restore
Custom threshold windows are saved as part of a full backup of your ITSI environment.
When you restore an ITSI backup, the status (Active, Scheduled, Completed, Restoring, or Skipped) for your custom threshold windows is updated based on the time that your environment is restored. For example, if your recurring custom threshold window was active during a backup job, but the scheduled time for your custom threshold window to execute has passed, then the custom threshold window's status and any linked KPIs are updated to Scheduled. If the custom threshold window was configured to run only once, then the status is updated Completed after restoring the backup.
Example custom threshold window use case
You want the ability to adjust the threshold configuration of your KPIs for large retail moments like a holiday weekend sale, because you notice abnormal changes in KPI values which generates false alerts that are only a result of this seasonal spike in website visits. Although adaptive thresholds can help to account for changes in your annual business cycle, they don't account for seasonal changes outside of the 60 day business cycle (the maximum training period supported for adaptive thresholds is 60 days), such as changes caused by seasonal holiday activity, and you also don't have enough annual data in order for adaptive thresholds to detect these special holidays. To solve this, you apply custom threshold windows to your KPIs, such as your KPI titled, Successful Checkouts. You set this window to run during the specific holiday time period so that your KPI threshold values are automatically adjusted and don't generate false alerts.
Create adaptive KPI thresholds in ITSI
Scenario: Apply adaptive thresholds to a KPI and detect outliers
This documentation applies to the following versions of Splunk® IT Service Intelligence: 4.15.0, 4.15.1, 4.15.2, 4.15.3, 4.16.0 Cloud only, 4.17.0, 4.17.1, 4.18.0