Modify notable event KV store collections in ITSI
IT Service Intelligence (ITSI) stores the status of notable events and episodes in KV store collections called itsi_notable_<object_type>. You can use SPL to repopulate or otherwise make bulk changes to the collections ITSI leverages for cached notable event state, typically referred to as the system and user lookups. Use the
outputlookup commands to perform CRUD operations on notable event KV store collection fields for maintenance and migration purpose without modifying configuration files.
If you only want to trim down notable event KV store collections without performing updates, you can do it through the itsi_notable_event_retention.conf file. This file lets you individually set retention limits for different collections, thereby deleting objects that have existed longer than the retention period. However, for a more dynamic and simpler solution, consider trimming them using SPL.
Guidelines for running the outputlookup command
By default the
outputlookup command overwrites your entire KV store collection. If you misuse the command, Event Analytics data will be lost and the contents of Episode Review will disappear. Follow these guidelines when running the
- Make sure to back up the KV store before running command in case you need to revert your changes.
- Check the results of the search before adding the
outputlookupline to make sure you're accessing the correct data.
|ITSI role||You must have the admin or itoa_admin ITSI role to modify notable event KV store collections.|
Modify collections using SPL
To modify notable event KV store collections using SPL, use an
inputlookup command to search the contents of a KV store lookup, then specify the operation to take on the lookup. If you're adding, modifying, or deleting entries, use
outputlookup to write the results to the KV store collection.
The following examples show different operations you can perform on KV store collections:
Check the episodes in the user collection generated in the last hour:
| inputlookup itsi_notable_group_user_lookup
| where mod_time > now() - 3600
Set the status of episodes older than two days to Resolved:
| inputlookup itsi_notable_group_user_lookup
| eval status = if(mod_time < now() - 2*24*3600, 4, status)
| outputlookup itsi_notable_group_user_lookup
Trim a collection to keep data from the last 90 days:
| inputlookup itsi_notable_group_system_lookup
| where mod_time > now() - 90*24*3600
| outputlookup itsi_notable_group_system_lookup
Trim collections using configuration files
By default, notable event metadata is archived after six months to keep the KV store from growing too large. If you have a large number of events, use the ITSI Health Check dashboard to check the collection sizes on disk and decide if you need to change the retention policy.
You can tune the retention policy for notable event metadata using an ITSI configuration file. The retention policy determines how long notable event metadata remains in the KV store before it is moved to
itsi_notable_archive. Retention policies are based on the mod_time (modify time), not the tag creation time.
- Only users with file system access, such as system administrators, can set the notable event retention policy.
- Review the steps in How to edit a configuration file in the Admin Manual.
Never change or copy the configuration files in the default directory. The files in the default directory must remain intact and in their original location.
- Open or create a local itsi_notable_event_retention.conf file in
- To set the retention time for all object types, reduce the
retentionTimeInSecin the [default] stanza. To individually set retention times for specific collections, add the
retentionTimeInSecsetting to individual stanzas as needed and reduce the retention time there. The settings in individual stanzas override what's set in the [default] stanza.
# default retention policy is 6 months retentionTimeInSec = 15768000 retentionObjectCount = 500000 disabled = 1 [itsi_notable_event_tag] # Check retention policy against mod_time (modify time of tags) and move tags to index once they meet the condition disabled = 0 object_type = notable_event_tag [itsi_notable_event_ticketing] # Check retention policy against mod_time (modify time of ticket) and move ticket to index once they meet the condition disabled = 0 object_type = external_ticket [itsi_notable_group_user] # Default is one year retentionTimeInSec = 31536000 retentionObjectCount = 500000 disabled = 0 object_type = notable_event_group [itsi_notable_group_system] # Default is one year retentionTimeInSec = 31536000 retentionObjectCount = 500000 disabled = 0 object_type = notable_group_system
Overview of notable events in ITSI
Clear all notable events in ITSI
This documentation applies to the following versions of Splunk® IT Service Intelligence: 4.8.0 Cloud only, 4.8.1 Cloud only, 4.9.0, 4.9.1, 4.9.2, 4.9.3, 4.10.0 Cloud only, 4.10.1 Cloud only, 4.10.2 Cloud only