Splunk® IT Service Intelligence

Administer Splunk IT Service Intelligence

Download manual as PDF

Download topic as PDF

Manage notable events in ITSI

Splunk IT Service Intelligence (ITSI) implements custom indexes for notable event storage. In a single instance deployment, the installation of ITSI creates the indexes in $SPLUNK_HOME/var/lib/splunk.

Index Description
itsi_tracked_alerts Stores active raw notable event data.
itsi_notable_audit Stores all audit events for episodes, including actions, comments, status change, and owner change.
itsi_grouped_alerts Stores active episode data.
itsi_notable_archive Stores episode tags that have been moved from the KV store after a default 6 month retention period, which begins when you close an episode in the UI. Moving data from the KV store removes extraneous data and helps improve performance.

ITSI uses an indexed real-time search to retrieve notable events from the Splunk platform. Indexed real-time searches have a delay of about 90 seconds before events get processed. Using concurrent real-time search instead of indexed real-time search is not supported for the itsi_event_grouping search because it significantly impacts system performance.

Set the notable event retention policy

In addition to the indexes listed above, ITSI stores the status of notable events and episodes in the KV store collections called itsi_notable_<object type>. By default, notable event metadata is archived after 6 months to keep the KV store from growing too large. If you have a large number of events, use the ITSI Health Check dashboard to check the collection sizes on disk and decide if you need to change the retention policy.

You can tune the retention policy for notable event metadata using an ITSI configuration file. The retention policy determines how long notable event metadata remains in the KV store before it is moved to itsi_notable_archive. Retention policies are based on the mod_time (modify time), not the tag creation time.

Prerequisites

  • Only users with file system access, such as system administrators, can set the notable event retention policy.
  • Review the steps in How to edit a configuration file in the Admin Manual.

Never change or copy the configuration files in the default directory. The files in the default directory must remain intact and in their original location.

Steps

  1. Open or create a local itsi_notable_event_retention.conf file in $SPLUNK_HOME/etc/apps/SA-ITOA/local/
  2. To set the retention time for all object types, reduce the retentionTimeInSec in the [default] stanza. To individually set retention times for specific object types, add the retentionTimeInSec setting to individual stanzas as needed and reduce the retention time there. The settings in individual stanzas override what's set in the [default] stanza.
# default retention policy is 6 months
retentionTimeInSec = 15768000
retentionObjectCount = 50000
disabled = 1

[itsi_notable_event_tag]
# Check retention policy against mod_time (modify time of tags) and move tags to index once they meet the condition
disabled = 0
object_type = notable_event_tag

[itsi_notable_event_ticketing]
# Check retention policy against mod_time (modify time of ticket) and move ticket to index once they meet the condition
disabled = 0
object_type = external_ticket

[itsi_notable_group_user]
# Default is one year
retentionTimeInSec = 31536000
retentionObjectCount = 50000
disabled = 0
object_type = notable_event_group

[itsi_notable_group_system]
# Default is one year
retentionTimeInSec = 31536000
retentionObjectCount = 50000
disabled = 0
object_type = notable_group_system

Backfill notable events into episodes

ITSI uses the itsi_event_grouping search command to aggregate notable events into episodes. If the search is disabled, either during a restart of Splunk Enterprise or manually by a user, ITSI does not group notable events into episodes.

When the itsi_event_grouping search is re-enabled, ITSI looks through notable events that were missed while the search was disabled and backfills them accordingly into episodes.

The default look-back time for missed events is 24 hours. If you want to change the look back time, modify the group_restore_lookback_time field in the itsi_rules_engine.properties file as follows.

Prerequisites

  • Only users with file system access, such as system administrators, can modify the look-back time for missed events using a properties file.
  • Review the steps in How to edit a configuration file in the Admin Manual.

Never change or copy the configuration files in the default directory. The files in the default directory must remain intact and in their original location.

Steps

  1. Open or create a local copy of the itsi_rules_engine.properties file at $SPLUNK_HOME/etc/apps/SA-ITOA/local/.
  2. Add the following setting to the file: group_restore_lookback_time = <number of hours>.

    For example, to look back two days, add the following setting:
    group_restore_lookback_time = 48
    

Clear all notable events

To permanently delete indexed notable events, use the CLI clean command. This command completely deletes the data in one or all indexes or KV store collections, depending on whether you provide an <index_name> or <collection> argument. For more information, see How to use the clean command in the Managing Indexers and Clusters of Indexers manual.

The clean command does not work on indexer clusters unless you run it separately on each indexer.

  1. In the CLI, to stop Splunk Enterprise type $SPLUNK_HOME/bin/splunk stop
  2. On each indexer, run the following commands to clear the indexes:
    $SPLUNK_HOME/bin/splunk clean eventdata -index itsi_tracked_alerts;
    $SPLUNK_HOME/bin/splunk clean eventdata -index itsi_notable_audit;
    $SPLUNK_HOME/bin/splunk clean eventdata -index itsi_notable_archive;
    $SPLUNK_HOME/bin/splunk clean eventdata -index itsi_grouped_alerts
    
  3. On a single search head, run the following commands to clear the KV store collections:
    $SPLUNK_HOME/bin/splunk clean kvstore -app SA-ITOA -collection itsi_notable_group_system;
    $SPLUNK_HOME/bin/splunk clean kvstore -app SA-ITOA -collection itsi_notable_group_user;
    $SPLUNK_HOME/bin/splunk clean kvstore -app SA-ITOA -collection itsi_notable_event_tag;
    $SPLUNK_HOME/bin/splunk clean kvstore -app SA-ITOA -collection itsi_notable_event_comment;
    $SPLUNK_HOME/bin/splunk clean kvstore -app SA-ITOA -collection itsi_notable_event_group;
    $SPLUNK_HOME/bin/splunk clean kvstore -app SA-ITOA -collection itsi_notable_event_actions_queue;
    $SPLUNK_HOME/bin/splunk clean kvstore -app SA-ITOA -collection itsi_temp_batch_claimed_action_queue;
    $SPLUNK_HOME/bin/splunk clean kvstore -app SA-ITOA -collection itsi_notable_event_ticketing
    
  4. To start Splunk Enterprise type $SPLUNK_HOME/bin/splunk start
PREVIOUS
Normalize event fields in ITSI
  NEXT
Tune notable event grouping in ITSI

This documentation applies to the following versions of Splunk® IT Service Intelligence: 4.4.1


Was this documentation topic helpful?

Enter your email address, and someone from the documentation team will respond to you:

Please provide your comments here. Ask a question or make a suggestion.

You must be logged into splunk.com in order to post comments. Log in now.

Please try to keep this discussion focused on the content covered in this documentation topic. If you have a more general question about Splunk functionality or are experiencing a difficulty with Splunk, consider posting a question to Splunkbase Answers.

0 out of 1000 Characters