Overview of the ITSI Rules Engine

The IT Service Intelligence (ITSI) Rules Engine is a system for continuously processing notable events to allow for event grouping and deduplication, as well as automatic action execution, based on user-defined criteria. The default system revolves around NATS, a queue-based messaging system that processes notable events and streams events directly to the Rules Engine. Starting with ITSI version 4.20.0, the NATS queue replaced the real-time search previously used by the Rules Engine to stream notable events from the itsi_tracked_alerts index.

A notable event aggregation policy (NEAP) is the fundamental unit of the Rules Engine. Aggregation policies are the data structure the Rules Engine uses to group notable events into episodes. It's also the container for action rules that automate episode actions, such as sending an email or pinging a host. For more information about aggregation policies, see Overview of aggregation policies in ITSI.

How the Rules Engine functions

The Rules Engine's functionality begins with correlation searches. Correlation searches generate notable events in ITSI, the NATS queue search processes event data immediately, and then sends this event data to the Rules Engine. Notable events are grouped into episodes based on grouping configurations defined by a notable event aggregation policy. Any actions associated with these notable events (also defined in a notable event aggregation policy) will also be executed immediately.

The Rules Engine pushes grouped alerts with the HTTP Event Collector (HEC) to the itsi_grouped_alerts index. The episode metadata is stored through REST in the itsi_notable_group_system and itsi_notable_group_user KV store collections. Event actions are pushed using REST in the itsi_notable_event_actions queue KV store collection, which are consumed by queue consumers.

The Rules Engine search periodically polls the itsi_notable_event_aggregation_policy KV store collection for updates. If a policy indicates some action should be executed, actions are pushed using a REST request in the itsi_notable_event_actions_queue KV store collection and dispatched to the queue consumers. For more information, see Event Management Interface in the REST API Reference manual.

To check on the health of the NATS queue or troubleshoot issues related to the queue, view the NATS Monitoring Dashboard.

Rules Engine queue mode

The Rules Engine groups notable events into episodes based on the filtering criteria you define in aggregation policies. Previously, a real-time search would stream all newly indexed events directly from the itsi_tracked_alerts index to the custom search command itsirulesengine. Starting with ITSI 4.20.0, notable events are streamed into the NATS queue from correlation searches, and then sent to the Rules Engine. NATS includes a built-in persistence engine called Jetstream that stores messages. You can access these messages in real-time, or store messages for later.

For more information about NATS and the Jetstream persistence engine, see the NATS Overview documentation.

Note: Events added to the itsi_tracked_alerts index using HEC won't be directly ingested to the NATS queue. Those notable events will be processed as missing events by the periodic backfill process, which runs every 12 minutes. To ingest these events, create a temporary index for these events, and use a correlation search to ingest these events into the Rules Engine running in queue mode, and to the itsi_tracked_alerts index.

Turn off queue mode

Prevent Rules Engine from continuing to run in queue mode by navigating to the Advanced Configuration page from the ITSI Configuration Assistant. Toggle off the setting for Rules engine queue mode.

Queue mode in an on-premises environment

For on-premises customers, port 4222 and port 4248 must be open between the search heads to allow NATS server peers to communicate over port 4248. Port 4222 is used to relay information to the NATS server. The correlation search produces notable events sent to the queue using port 4222. The Rules Engine ingests notable events from the queue using port 4222.

Rules Engine real-time search

The Rules Engine used the itsi_event_grouping real-time search until ITSI version 4.19.0, and the search was then replaced by a queue-based messaging system powered by NATS connective technology.

The Rules Engine real time search groups notable events into episodes based on the filtering criteria defined in aggregation policies. The search is similar to the following:

search `itsi_tracked_alerts_index` | itsi_rules_engine | where 1=2

The search runs with a time range of earliest=rt, latest=rt. It includes indexedRealTime=1 to force Splunk to stream all newly indexed events directly to the custom search command itsi_rules_engine. The final WHERE clause ensures the output of the search command doesn't linger in the dispatch directory.

Sort notable events with the Rules Engine

When incoming notable events are generated within a few seconds of each other and you want to process them according to the event time stamp, update the Rules Engine using these steps:

If you are using correlation searches to generate notable events, update the following:

1. Edit $SPLUNK_HOME/etc/apps/SA-ITOA/local/itsi_rules_engine.properties and add: rules_engine_feature_disabled_list = POLICY_EXECUTOR_ASYNC_SUB_ACTORS, POLICY_EXECUTOR_STATE_RECOVERY
2. Edit $SPLUNK_HOME/etc/apps/SA-ITOA/local/itsi_event_management.conf and add: [tracked_alert] sort_notable_events = 1
3. Edit $SPLUNK_HOME/etc/apps/SA-ITOA/local/alert_actions.conf and add: [itsi_event_generator] param.is_use_event_time = 1
4. Restart Splunk.

If you are directly ingesting notable events through HEC, update the following:

1. Ensure that the events are ingested in a sorted manner according to the timestamp in an ascending order.
2. Edit $SPLUNK_HOME/etc/apps/SA-ITOA/local/itsi_rules_engine.properties and add: rules_engine_feature_disabled_list = POLICY_EXECUTOR_ASYNC_SUB_ACTORS, POLICY_EXECUTOR_STATE_RECOVERY
3. Restart the Rules Engine.
4. Navigate to Activity, then Jobs, then Set all filters to All, then search for label="itsi_event_grouping", then stop the job. Wait for a few minutes to Rules Engine to run again.