
Manage notable event indexes in ITSI
Splunk IT Service Intelligence implements custom indexes for notable event storage. In a single instance deployment, the installation of ITSI creates the indexes in $SPLUNK_HOME/var/lib/splunk
.
Index | Description |
---|---|
itsi_tracked_alerts | Stores active raw notable event data. |
itsi_notable_audit | Stores all audit events for episodes, including actions, comments, status change, and owner change. |
itsi_grouped_alerts | Stores active episode data. |
itsi_notable_archive | Stores episode metadata (tags and comments) that has been moved from the KV store after a default 6 month retention period, which begins when you close an episode in the UI. Moving data from the KV store removes extraneous data and helps improve performance. |
Set the notable event retention policy
In addition to the indexes listed above, ITSI stores the status of notable events and episodes in the KV store collections called itsi_notable_<object type>
. By default, notable event metadata is archived after 6 months to keep the KV store from growing too large. If you have a large number of events, use the ITSI Health Check dashboard to check the collection sizes on disk and decide if you need to change the retention policy.
You can tune the retention policy for notable event metadata using an ITSI configuration file. The retention policy determines how long notable event metadata remains in the KV store before it is moved to itsi_notable_archive
. Retention policies are based on the mod_time (modify time), not the tag or comment creation time.
Steps
- Edit
$SPLUNK_HOME/etc/apps/SA-ITOA/local/itsi_notable_event_retention.conf
- In the
[default]
stanza, and in individual stanzas as needed, setretentionTimeInSec
to a new value.
[default] # default retention policy is 6 months retentionTimeInSec = 15768000 disabled = 1 [itsi_notable_event_tag] # Check retention policy against mod_time (modify time of tags) and move tags to index once they met condition disabled = 0 object_type = notable_event_tag [itsi_notable_event_comment] # Check retention policy against mod_time (modify time of comment) and move comment to index once they met condition disabled = 0 object_type = notable_event_comment [itsi_notable_event_ticketing] # Check retention policy against mod_time (modify time of ticket) and move ticket to index once they met condition disabled = 0 object_type = external_ticket [itsi_notable_event_group] # Default is one year retentionTimeInSec = 31536000 disabled = 0 object_type = notable_event_group [itsi_notable_event_state] # Default is one year retentionTimeInSec = 31536000 disabled = 0 object_type = notable_event_state [itsi_notable_group_user] # Default is one year retentionTimeInSec = 31536000 disabled = 0 object_type = notable_event_group [itsi_notable_group_system] # Default is one year retentionTimeInSec = 31536000 disabled = 0 object_type = notable_group_system
Use notable event real-time search optimization
ITSI uses an indexed real-time search to retrieve notable events from the Splunk platform. Indexed real-time searches have a delay of about 90 seconds before events get processed. If you require faster processing times, you can change the ITSI search to use a real-time search instead of an indexed real-time search.
Changing from indexed real-time search to real-time search significantly increases the requirements of running the system and will impact the speed of indexing.
Change the search to real-time by modifying savedsearches.conf
:
- Edit
$SPLUNK_HOME/etc/apps/SA-ITOA/local/savedsearches.conf
- In the
[itsi_event_grouping]
stanza, setdispatch.indexedRealtime
to 0.
[itsi_event_grouping] dispatch.indexedRealtime = 0
Backfill notable events into episodes
ITSI uses the itsi_event_grouping
search command to aggregate notable events into episodes. If this search is disabled, ITSI will not group notable events into episodes. This can occur during a restart of Splunk Enterprise, or if a user manually disables the search.
When the itsi_event_grouping
search is re-enabled, ITSI looks through notable events that were missed while the search was disabled, and backfills them accordingly into episodes.
The default look back time for missed events is 24 hours. If you want to change the look back time, modify the group_restore_lookback_time
field in the itsi_rules_engine.properties
file, as follows:
- Create a local version of
itsi_rules_engine.properties
in$SPLUNK_HOME/etc/apps/SA-ITOA/local/itsi_rules_engine.properties
. - Add the following parameter to the file:
group_restore_lookback_time=<number of hours>
.
For example, to look back 2 days, add the following:group_restore_lookback_time=48
- Create a local version of
commands.conf
inSA-ITOA/local/commands.conf
. Add the following lines to point to the local version ofitsi_rules_engine.properties
:
[itsirulesengine] command.arg.3=-DitsiRulesEngine.configurationFile=../local/itsi_rules_engine.properties.
Clear all notable events
To permanently delete indexed notable events, use the CLI clean
command. This command completely deletes the data in one or all indexes or KV store collections, depending on whether you provide an <index_name>
or <collection>
argument. For more information, see How to use the clean command in the Managing Indexers and Clusters of Indexers manual.
The clean
command does not work on indexer clusters unless you run the clean command separately on each indexer.
- In the CLI, to stop Splunk Enterprise type
$SPLUNK_HOME/bin/splunk stop
- On each indexer, run the following commands to clear the indexes:
$SPLUNK_HOME/bin/splunk clean eventdata -index itsi_tracked_alerts; $SPLUNK_HOME/bin/splunk clean eventdata -index itsi_notable_audit; $SPLUNK_HOME/bin/splunk clean eventdata -index itsi_notable_archive; $SPLUNK_HOME/bin/splunk clean eventdata -index itsi_grouped_alerts
- On a single search head, run the following commands to clear the KV store collections:
$SPLUNK_HOME/bin/splunk clean kvstore -app SA-ITOA -collection itsi_notable_group_system; $SPLUNK_HOME/bin/splunk clean kvstore -app SA-ITOA -collection itsi_notable_group_user; $SPLUNK_HOME/bin/splunk clean kvstore -app SA-ITOA -collection itsi_notable_event_tag; $SPLUNK_HOME/bin/splunk clean kvstore -app SA-ITOA -collection itsi_notable_event_comment; $SPLUNK_HOME/bin/splunk clean kvstore -app SA-ITOA -collection itsi_notable_event_group; $SPLUNK_HOME/bin/splunk clean kvstore -app SA-ITOA -collection itsi_notable_event_actions_queue; $SPLUNK_HOME/bin/splunk clean kvstore -app SA-ITOA -collection itsi_temp_batch_claimed_action_queue; $SPLUNK_HOME/bin/splunk clean kvstore -app SA-ITOA -collection itsi_notable_event_ticketing
- To start Splunk Enterprise type
$SPLUNK_HOME/bin/splunk start
PREVIOUS ITSI Predictive Analytics use case |
NEXT Set up custom episode actions in ITSI |
This documentation applies to the following versions of Splunk® IT Service Intelligence: 4.0.0, 4.0.1, 4.0.2, 4.0.3, 4.0.4
Feedback submitted, thanks!