Splunk® IT Service Intelligence

Event Analytics Manual

Download manual as PDF

Download topic as PDF

Troubleshoot the Rules Engine and event grouping in ITSI

Here are some common issues with notable event grouping in IT Service Intelligence (ITSI) and how to resolve them.

Issue: Events aren't being grouped into episodes

Even though notable events are coming into ITSI, aggregation policies aren't grouping them into episodes.

Mitigation

1. First, go to Configuration > Notable Event Aggregation Policies within ITSI and make sure you have aggregation policies created and that they're enabled.

2. Make sure the Rules Engine is running. Go to Activity > Jobs and change the App context to All. Search for itsi_event_grouping and make sure the status says "Running". If it's not running, go to Settings > Searches, reports, and alerts and change the app context to All. Search for the itsi_event_grouping search again and check that it's enabled. If not, enable it.

3. Make sure you have the correct Java version. See Java requirements in the Install and Upgrade manual.

4. Make sure the itsi_grouped_alerts index exists and contains data. The index is deployed in the SA-IndexCreation directory.

5. If the Rules Engine is running, run the following search to check for error messages that might indicate what's causing the problem:

index=_internal source=*rules_engine* log_level=ERROR

6. If the above steps fail, change the log level to debug at $SPLUNK_HOME/etc/apps/SA-ITOA/default/log4j_rules_engine.xml and finalize the old itsi_event_grouping search so a new one can pick up updated log level information:

<root level= "debug" >
     <!--<appender-ref ref= "Console" /> <!– To console –>-->
     <appender-ref ref= "RollingFile" /> <!-- And to a rotated file -->
</root>

7. If the problem persists, file a ticket with Splunk Support.

Issue: The Rules engine search (| itsirulesengine) is failing

You see the following error message:

Exception while parsing metadata JSON: Unexpected character in string: '\0A'

Mitigation

  1. Update your Java version to 1.8 or higher. See Java requirements in the Install and Upgrade manual.
  2. Set the JAVA_HOME environment variable to the new Java version.
  3. Restart your Splunk software.

Issue: Events are being grouped by the default aggregation policy instead of a custom policy

Even though notable events are being ingested into ITSI, they aren't being grouped by any of the custom aggregation policies you've created.

Mitigation

Events are only grouped by the default aggregation policy when they don't match the filtering criteria of any existing policies. Check the filtering criteria of the policies you've created. For more information, see Configure episode filtering and breaking criteria in ITSI.

Events are added to new groups instead of to existing active groups

The sub-group limit (sub_group_limit) has been exceeded, so the hash keys and the episodes associated with them are cleared from memory. Any new events with the old hash key create new sub-groups, which causes new episode to be created.

Mitigation

  1. Create a local copy of itsi_rules_engine.properties at $SPLUNK_HOME/etc/apps/SA-ITOA/local
  2. Increase the sub_group_limit setting accordingly.

For more information about sub-groups and other sizing parameters, see Tune episode and aggregation policy sizing parameters in ITSI.

Issue: "Connection Refused" error

You see the following error message:

error=Connection refused java.lang.RuntimeException: Connection refused

Mitigation

Restart the Rules Engine, as this could mean Splunkd wasn't set up properly:

  1. Within Splunk Web, go to Settings > Searches, reports, and alerts.
  2. In the App dropdown, select All.
  3. Use the filter to locate the itsi_event_grouping search.
  4. Click Actions > Disable.
  5. Wait for about 10 seconds, then re-enable it again.

If the restart doesn't solve the problem, check your network permissions.

Issue: Notable event KV store collections are growing very large

This issue occurs because the indexed realtime search returns events over and over from buckets that use tsidx reduction.

Mitigation

Disable tsidx reduction on the itsi_tracked_alerts and itsi_summary indexes and rebuild all old buckets on these indexes. For more information, see Reduce tsidx disk usage in the Managing Indexers and Clusters of Indexers manual.

Upon upgrade, the Rules Engine search command fails

You see the following error message:

Error occurred during initialization of VM

Mitigation

This issue occurs because 32-bit Java can't run the Rules Engine with the new memory settings introduced in version 4.3.x.

  1. Open or create a local copy of commands.conf at $SPLUNK_HOME/etc/apps/SA-ITOA/local.
  2. Add the following stanza:
    [itsirulesengine]
     command.arg.1=-J-Xmx1024M
     # reduced to 1024MB for 32 bit JDK/JRE
  3. Restart the Rules Engine, either by disabling and reenabling the itsi_event_grouping search, or by restarting your Splunk software.

ITSI is constantly using 100% CPU because multiple java.exe processes are running

This issue occurs because ITSI Event Analytics is incompatible with Splunk Enterprise versions 7.2.4 - 7.2.10.

Mitigation

Perform the workaround in SPL-155648.

Last modified on 22 September, 2020
PREVIOUS
Best practices for implementing Event Analytics in ITSI
  NEXT
Configure the Rules Engine to handle indexer cluster rolling restarts and upgrades

This documentation applies to the following versions of Splunk® IT Service Intelligence: 4.5.0 Cloud only, 4.5.1 Cloud only, 4.6.0 Cloud only, 4.6.1 Cloud only, 4.6.2 Cloud only


Was this documentation topic helpful?

Enter your email address, and someone from the documentation team will respond to you:

Please provide your comments here. Ask a question or make a suggestion.

You must be logged into splunk.com in order to post comments. Log in now.

Please try to keep this discussion focused on the content covered in this documentation topic. If you have a more general question about Splunk functionality or are experiencing a difficulty with Splunk, consider posting a question to Splunkbase Answers.

0 out of 1000 Characters