Troubleshoot the Rules Engine and event grouping in ITSI
Here are some common issues with notable event grouping in IT Service Intelligence (ITSI) and how to resolve them.
Events aren't being grouped into episodes
Even though notable events are coming into ITSI, aggregation policies aren't grouping them into episodes.
1. First, go to Configuration > Notable Event Aggregation Policies within ITSI and make sure aggregation policies are created and enabled.
2. Make sure the Rules Engine is running. Go to Activity > Jobs and change the App context to All. Search for itsi_event_grouping and make sure the status says Running. If it's not running, go to Settings > Searches, reports, and alerts and change the app context to All. Search for the itsi_event_grouping search again and check that it's enabled. If not, enable it.
3. Make sure you have the correct Java version. See Java requirements in the Install and Upgrade manual.
4. Make sure the itsi_grouped_alerts index exists and contains data. The index is deployed in the SA-IndexCreation directory.
5. If the Rules Engine is running, run the following search to check for error messages that might indicate what's causing the problem:
index=_internal source=*rules_engine* log_level=ERROR
6. If the previous steps fail, change the log level to debug at
$SPLUNK_HOME/etc/apps/SA-ITOA/default/log4j_rules_engine.xml and finalize the old itsi_event_grouping search so a new one can pick up updated log level information:
<root level= "debug" > <!--<appender-ref ref= "Console" /> <!– To console –>--> <appender-ref ref= "RollingFile" /> <!-- And to a rotated file --> </root>
7. If the problem persists, file a ticket with Splunk Support.
The Rules engine search is failing
You see the following error message:
Exception while parsing metadata JSON: Unexpected character in string: '\0A'
- Update your Java version to 1.8 or higher. See Java requirements in the Install and Upgrade manual.
- Set the JAVA_HOME environment variable to the new Java version.
- Restart your Splunk software.
Events are being grouped by the default aggregation policy instead of a custom policy
Even though notable events are being ingested into ITSI, they aren't being grouped by any of the custom aggregation policies you've created.
Events are only grouped by the default aggregation policy when they don't match the filtering criteria of any existing policies. Check the filtering criteria of the policies you've created. For more information, see Configure episode filtering and breaking criteria in ITSI.
Events are added to new episodes instead of to existing active episodes
The sub-group limit (
sub_group_limit) has been exceeded, so the hash keys and the episodes associated with them are cleared from memory. Any new events with the old hash key create new sub-groups, which causes new episode to be created.
- Create a local copy of
- Increase the
For more information about sub-groups and other sizing parameters, see Tune episode and aggregation policy sizing parameters in ITSI.
Java process not starting when the Rules Engine search is executed
The Java process isn't starting when the
itsirulesengine command is executed and no itsi_rules_engine.log files are generated.
- If you see the following error message:
Invalid message received from external search command during setup, see search.log
verify that /opt/splunk/etc/apps/SA-ITOA/bin/itsirulesengine has execute permissions. Run the following command from /opt/splunk/etc/apps/SA-ITOA/bin to see if you get a permission denied error:
/opt/splunk/etc/apps/SA-ITOA/bin/itsirulesengine -J-Xmx2048M -Dlog4j.configurationFile=../default/log4j_rules_engine.xml -DitsiRulesEngine.configurationFile=../default/itsi_rules_engine.properties -Dfile.encoding=UTF-8 -Dconfig.file=../default/akka_application.conf
- Make sure no duplicate JAR files are present under /opt/splunk/etc/apps/SA-ITOA/lib/java/event_management/libs and that all the JAR files have execute permission for the Splunk user.
- Make sure your Java is not 32-bit JRE/JDK. For more information, see Java requirements in the Install and Upgrade Manual.
- Make sure
rtsearch, the real-time search capability, isn't disabled in the
[role_admin]stanza in $SPLUNK_HOME/etc/apps/itsi/default/authorize.conf.
"Connection Refused" error
You see the following error message:
error=Connection refused java.lang.RuntimeException: Connection refused
Restart the Rules Engine, as this could mean Splunkd wasn't set up properly:
- Within Splunk Web, go to Settings > Searches, reports, and alerts.
- In the App dropdown, select All.
- Use the filter to locate the itsi_event_grouping search.
- Click Actions > Disable.
- Wait for about 10 seconds, then re-enable it again.
If the restart doesn't solve the problem, check your network permissions.
Notable event KV store collections are growing very large
This issue occurs because the indexed realtime search returns events over and over from buckets that use tsidx reduction.
Disable tsidx reduction on the itsi_tracked_alerts and itsi_summary indexes and rebuild all old buckets on these indexes. For more information, see Reduce tsidx disk usage in the Managing Indexers and Clusters of Indexers manual.
Upon upgrade, the Rules Engine search command fails
You see the following error message:
Error occurred during initialization of VM
This issue occurs because 32-bit Java can't run the Rules Engine with the new memory settings introduced in version 4.3.x.
- Open or create a local copy of
- Add the following stanza:
[itsirulesengine] command.arg.1=-J-Xmx1024M # reduced to 1024MB for 32 bit JDK/JRE
- Restart the Rules Engine, either by disabling and reenabling the
itsi_event_groupingsearch, or by restarting your Splunk software.
ITSI is constantly using 100% CPU because multiple java.exe processes are running
This issue occurs because ITSI Event Analytics is incompatible with Splunk Enterprise versions 7.2.4 - 7.2.10.
Perform the workaround in SPL-155648.
Notable Event Aggregation Policy filter doesn't work with special characters
Rules Engine errors occur if the filter criteria contains special characters like
\ character before each of the special characters.
Some search peers aren't contacted when realtime search runs
The realtime search itsi_event_grouping misses some events. If the there is delay in grouping, backfilling, or missing events, it could be because of this issue.
Compare the number of active peers against the number of hosts identified using steps below. If the number of hosts decreases over time, then it's likely that not all peers are being contacted. This data is correlated in time against the volume of skipped events.
- Run the following search to determine the number of peers participating in the itsi_event_grouping search over time:
index=_internal stack::colgateg sourcetype=splunk_resource_usage host=idx-* data.search_props.label=itsi_event_grouping | timechart dc(host)
- Determine how many indexers are active by selecting Settings > Indexer Cluster.
- Perform a rolling restart on the stack, or restart the itsi_event_grouping search on the ITSI search head. After the search has been terminated, the scheduler will quickly restart it.
There are indexing delays
There could be multiple different causes of indexing delays and different methods to check for indexing delays. If the there is delay in grouping, backfilling, or missing events, it could be because of this issue.
Step 1: Detect indexing delays
You can detect index delays using one of the following methods:
Method 1: Check difference between _indextime and _time of the event
- Run the following search to check indexing delays:
index=<indexname> | eval delay=_indextime-_time | stats min(delay) avg(delay) max(delay)
- If is_use_event_time is enabled, decide what _indextime should be based on how often the correlation search runs, and when the event was originally indexed.
- Check for delays between _indextime and _time. A delay longer than 30 seconds indicates a problem.
For the itsi_tracked_alerts index, it's not possible to check indexing delays unless the is_use_event_time is enabled for itsi_event_generator in SA-ITOA/local/alert_actions.conf. If enabled, the _time and org_time of the events retains the org_time of the event; otherwise, these fields are replaced by the _indextime.
Method 2: Check CPU raw data write usage
You can can look at Splunk Monitoring Console to check the CPU raw data write usage. If CPU raw data write usage is higher than the time Splunk spends processing events, then there are indexing delays.
- Select Indexing in Monitoring Console.
- Select Indexing Performance > Indexing Performance: Instance.
- Check the Cumulative CPU seconds spent per indexer processor activity panel on the dashboard, which breaks down the utilization metrics for raw data write and the indexing service on an indexer.
Step 2: Tune the index time-realtime offset
By default the indextime-realtime searches use the default 60 second offset setting. This offset can be tuned for specific searches, but will add some latency on the realtime results. Adjust the offset until you have fewer backfilled events:
- On the ITSI search head, select Settings > Saved searches and alerts > "itsi_event_grouping" on the SA-ITOA app.
- Select the Advanced edit setting. Update the value of the dispatch.indexedRealtimeOffset setting to a value greater than 60 seconds.
- Disable, then re-enable the saved search itsi_event_grouping and force the search to restart.
- Use the ITSI Events Analytics Monitoring dashboard to compare the new settings to the previous settings using the Backfilled events percentage panel. For more information about the Event Analytics Monitoring dashboard, see Event Analytics Monitoring dashboard.
You may have to continue updating the value of the setting with different values, such as 60 to 90, to 120, until you reduce the number of backfilled events drops.
Search peers won't connect and cause a delay in grouping events
If the there is delay in grouping, backfilling, or missing events, it could be because of this issue.
Use the following command to identify if there are issues with the search peers and if the results match the timeline of missing or delayed events:
index="_internal" source="*splunkd*" "unable to distribute to the peer" OR "might have returned partial results" OR "search results might be incomplete" OR "search results may be incomplete" OR "unable to distribute to peer" OR "search process did not exit cleanly" OR "streamed search execute failed" OR "failed to create a bundles setup with server"
If issues continue, contact Splunk Support.
Best practices for implementing Event Analytics in ITSI
Configure the Rules Engine to handle indexer cluster rolling restarts and upgrades
This documentation applies to the following versions of Splunk® IT Service Intelligence: 4.11.1, 4.11.2, 4.11.3, 4.11.4, 4.11.5, 4.11.6, 4.12.0 Cloud only, 4.12.1 Cloud only, 4.12.2 Cloud only, 4.13.0, 4.13.1, 4.13.2, 4.14.0 Cloud only, 4.14.1 Cloud only, 4.14.2 Cloud only, 4.15.0, 4.16.0 Cloud only