Event Analytics Monitoring dashboard

The Event Analytics Monitoring dashboard provides troubleshooting information for ITSI's Event Analytics functionality.

Dashboard panels

Panel	Description
Rules Engine Information	The Java version being used by the Rules Engine. ITSI requires Java 8 - 11 to run notable event management features.
Rules Engine Event Processing Stats	The number of events processed by each Rules Engine activity every 10 minutes.
Rules Engine Schedulers Health Check	The number of times various Rules Engine schedulers run per 12-minute time frame. These statistics are used internally to ensure the schedulers are running as expected. By default, the Event Periodic Backfill Scheduler runs every 12 minutes, the Policy Rules Check Scheduler runs every 1 minute for each aggregation policy, and the Policy Group Updates KV Store Sync Scheduler runs every 28 seconds for each aggregation policy.
Rules Engine Starts and Stops	The number of times the Rules Engine starts and stops each hour. A Rules Engine restart can kick off multiple backfill processes with the default phased_execution_mode value in limits.conf, which might lead to the creation of duplicate episodes. Restarts can also help troubleshoot if you aren't seeing certain expected episodes.
Rules Engine Activity	The number of states for each completed Rules Engine instance. You can use the instance ID to search the ITSI logs and troubleshoot issues.
Rules Engine Activity Details	The details of activities of each Rules Engine instance completed. You can use the instance ID to search the ITSI logs and troubleshoot issues.
Skipped Events Count	A raw count of skipped events (events that are not included in any episodes) over the past 7 days. Under normal conditions, this number should be zero.
Skipped Events Percentage	The percentage of ungrouped events versus grouped events over the past 7 days. Under normal conditions, this percentage should be zero.
Episode Processing Times	The amount of time it takes to convert tracked alerts (active raw notable events) to grouped alerts (active grouped notable events). Under normal conditions, the processing time should be about 60 seconds.
Event Processing Times	The median time each Rules Engine component takes to process events. This time does not include the real-time search delay and is calculated from the point at which event is received by the Rules Engine.
Event Volume	The number of events tracked in its_tracked_alerts, processed by the Rules Engine, and ingested into its_grouped_alerts per 10-minute time frame. Use this panel to troubleshoot grouping issues.
Event Analytics Real-Time Search Status	The current state of real-time searches, including how much disk space they've used so far and how long they've been running. The searches exist in savesearches.conf.
Event Analytics HEC Tokens	Shows which HEC tokens are available by host. If you create notable events using HEC tokens, this table shows which of your instances to send events to using the 'Auto Generated ITSI Event Management Token'. The absence of any of these tokens will lead to event analytics not working properly.
Event Analytics KV Store Lookups	Compares the created KV store lookups with the ones that are required for event analytics but not created. If a lookup is not created, you must add it to transforms.conf.
Event Analytics Action Queue Errors	A count of action queue errors over time. To search for the action queue errors, run the following search: index=_internal sourcetype="itsi_internal_log" source="*itsi_notable_event_actions_queue_consumer*" ERROR
Notable Event Size Check	Notable event sizes over time. The maximum allowable event size is 10000 bytes. If your events exceed this limit, increase the TRUNCATE setting in props.conf.
Events By Correlation Searches and Indexes	The number of tracked alerts and grouped alerts per correlation search. Use the dropdown menu to filter by individual correlation searches.
Events by Aggregation Policy	The number of grouped alerts per aggregation policy. You can filter by one or more aggregation policies to compare the number of events per policy.