Modify notable event KV store collections in ITSI

IT Service Intelligence (ITSI) stores the status of notable events and episodes in KV store collections called itsi_notable_<object_type>. You can use SPL to repopulate or otherwise make bulk changes to the collections ITSI leverages for cached notable event state, typically referred to as the system and user lookups. Use the inputlookup and outputlookup commands to perform CRUD operations on notable event KV store collection fields for maintenance and migration purposes without modifying configuration files.

If you only want to trim down notable event KV store collections without performing updates, you can do it through the itsi_notable_event_retention.conf file. This file lets you set individual retention limits for different collections, thereby deleting objects that have existed longer than the retention period. However, for a more dynamic and simpler solution, consider trimming notable event KV store collections using SPL.

Guidelines for running the outputlookup command

By default the outputlookup command overwrites your entire KV store collection. If you misuse the command, Event Analytics data will be lost and the contents of Episode Review will disappear. Follow these guidelines when running the outputlookup command:

• Make sure to back up the KV store before running command in case you need to revert your changes.
• Check the results of the search before adding the outputlookup line to make sure you're accessing the correct data.
• Stop the itsi_event_grouping search before using the outputlookup command with itsi_notable_group collections so as to ensure that collections remain in sync with each other.

If you decide to modify notable event and episode KV store collections independently of each other, keep episodes manageable by making sure that any modification to one collection has a corresponding record in the other collection.

Prerequisite

Requirement	Description
ITSI role	You must have the admin or itoa_admin ITSI role to modify notable event KV store collections.

Modify collections using SPL

To modify notable event KV store collections using SPL, use an inputlookup command to search the contents of a KV store lookup, then specify the operation to take on the lookup. If you're adding, modifying, or deleting entries, use outputlookup to write the results to the KV store collection.

The following examples show different operations you can perform on KV store collections:

Read

Check the episodes in the user collection generated in the last hour:

| inputlookup itsi_notable_group_user_lookup | where mod_time > now() - 3600

Update

Set the status of episodes older than two days to Resolved:

| inputlookup itsi_notable_group_user_lookup | eval status = if(mod_time < now() - 2*24*3600, 4, status) | outputlookup itsi_notable_group_user_lookup

Delete

Trim a collection to keep data from the last 90 days:

| inputlookup itsi_notable_group_system_lookup | where mod_time > now() - 90*24*3600 | outputlookup itsi_notable_group_system_lookup

Trim collections using configuration files

By default, notable event metadata is archived after six months to keep the KV store from growing too large. If you have a large number of events, use the ITSI Health Check dashboard to check the collection sizes on disk and decide if you need to change the retention policy.

You can tune the retention policy for notable event metadata using an ITSI configuration file. The retention policy determines how long notable event metadata remains in the KV store before it is moved to itsi_notable_archive. Retention policies are based on the mod_time (modify time), not the tag creation time.

Notable events linked to open episodes don't get archived. However, if all the episodes are open, the oldest open episodes are archived when the retentionTimeInSec or retentionObjectCount is reached.

Prerequisites

• Only users with file system access, such as system administrators, can set the notable event retention policy.
• Review the steps in How to edit a configuration file in the Admin Manual.

Never change or copy the configuration files in the default directory. The files in the default directory must remain intact and in their original location.

Steps

1. Open or create a local itsi_notable_event_retention.conf file in $SPLUNK_HOME/etc/apps/SA-ITOA/local/
2. To set the retention time for all object types, reduce the retentionTimeInSec in the [default] stanza. To individually set retention times for specific collections, add the retentionTimeInSec setting to individual stanzas as needed and reduce the retention time there. The settings in individual stanzas override what's set in the [default] stanza.
   

   # default retention policy is 6 months
   retentionTimeInSec = 15768000
   retentionObjectCount = 500000
   disabled = 1
   
   [itsi_notable_event_tag]
   # Check retention policy against mod_time (modify time of tags) and move tags to index once they meet the condition
   disabled = 0
   object_type = notable_event_tag
   
   [itsi_notable_event_ticketing]
   # Check retention policy against mod_time (modify time of ticket) and move ticket to index once they meet the condition
   disabled = 0
   object_type = external_ticket
   
   [itsi_notable_group_user]
   # Default is one year
   retentionTimeInSec = 31536000
   retentionObjectCount = 500000
   disabled = 0
   object_type = notable_event_group
   
   [itsi_notable_group_system]
   # Default is one year
   retentionTimeInSec = 31536000
   retentionObjectCount = 500000
   disabled = 0
   object_type = notable_group_system