Known issues in Splunk IT Service Intelligence

IT Service Intelligence (ITSI) version 4.11.3 has the following known issues and workarounds.

Adaptive Thresholding

Date filed	Issue number	Description
2022-02-10	ITSI-21921	Preview for aggregate thresholds takes a long time to load Workaround: From the Preferences modal, set the Time zone to the Default System Timezone.
2021-11-05	ITSI-19663	Updating a KPI threshold policy within a service template causes the thresholds of all existing KPIs that use adaptive thresholds to get reset Workaround: Temporary workaround to avoid false alerts: # Put services that are linked to the service template into maintenance mode # Make KPI threshold changes within the service template and push out # Wait to make sure all services are synced # Manually run the itsi_at_search_kpi_minusXd to recreate the adaptive threshold models # Disable maintenance mode for false alerts

Backup/Restore and Migration Issues

Date filed	Issue number	Description
2021-12-06	ITSI-20325	When a backup .ZIP file includes a base search with a title that is over 100 characters, the backup restore job will fail.
2021-12-02	ITSI-20308	Errors found in the migration log while upgrading to 4.11.0
2021-10-13	ITSI-19215	Customer is getting a lot of errors related to "Could not find object id=itsi_entity_dashboard_drilldown" after installation of IT Essentials Work Workaround: Upgrade to ITE-Work version 4.12 and later

Deep Dive

Date filed	Issue number	Description
2022-05-19	ITSI-24186	Auto save for a default deep dive is not working.

Notable Events

Date filed	Issue number	Description
2022-12-20	ITSI-27751	Episode Review arbitrary search filter with AND & OR conditions fail to match events under certain scenarios Workaround: Avoid using brackets (), extra whitespaces, the operator !=, and double quotes "" in the search filter
2022-01-25	ITSI-21269	The grouping of Bidirectional Ticketing events sets the episode KV store state to faulty values
2022-01-05	ITSI-20978	ServiceNow bidirectional integration resets the episode title, status, severity, and owner. Workaround: #In etc/app/itsi/local/savedsearches.conf, replace the Bidirectional Ticketing clause with the following: \[Bidirectional Ticketing] action.itsi_event_generator.param.description = %group_description% action.itsi_event_generator.param.itsi_instruction = %group_instruction% action.itsi_event_generator.param.owner = %group_assignee% action.itsi_event_generator.param.severity = %group_severity% action.itsi_event_generator.param.status = %group_status% action.itsi_event_generator.param.title = %group_title% disabled = 0 dispatch.earliest_time = -4h search = | `itsi_bidirectional_ticketing(main,90,itsi_tracked_alerts)`\\ | join group_id\\     \[ search `itsi_event_management_group_index_with_state("")` | rename itsi_group_id as group_id | stats latest(itsi_group_title) as group_title latest(severity) as group_severity latest(owner) as group_assignee latest(itsi_group_description) as group_description latest(itsi_group_instruction) as group_instruction latest(status) as group_status by group_id] If any NEAP action rules has the following condition Create SNOW ticket when number of events is equal to 1, there is a chance that the issue will occasionally come up due to a race condition between the initialization of the episode and the creation of the SNOW ticket. Change the activation criteria to when number of events is equal to 2 (or anything else) Increase action_execution_delay in itsi_rules_engine.properties to a higher value (i.e. 100) Restart Splunk.
2021-12-07	ITSI-20343	Impacted Services and KPIs do not appear in Episode Review when using Teams functionality Workaround: Create/edit Template:SA-ITOA/local/macros.conf and add the following two stanzas: {noformat}# Take in a string of team_keys in the format of '(sec_grp="itsi_team_key") OR (sec_grp="itsi_team_key")' and returns a filter of service_ids [itsi_events_compare_teams(1)] args = itsi_team_id_list definition = search (service_ids=*null*) OR (NOT service_ids=*) OR [|inputlookup itsi_services_in_team_lookup where ($itsi_team_id_list$) | rename _key as service_ids | eval service_ids="*".service_ids."*" | fields service_ids] Take in a string of team_keys in the format of '(sec_grp="itsi_team_key") OR (sec_grp="itsi_team_key")' and returns a filter of itsi_service_ids [itsi_groups_compare_teams(1)] args = itsi_team_id_list definition = search (itsi_service_ids=*null*) OR (NOT itsi_service_ids=*) OR [|inputlookup itsi_services_in_team_lookup where ($itsi_team_id_list$) | rename _key as itsi_service_ids | eval itsi_service_ids="*".itsi_service_ids."*" | fields itsi_service_ids]{noformat}
2021-12-03	ITSI-20314	Episode not being marked as inactive when bulk close is used
2021-10-20	ITSI-19415	On Windows server, more than 1 rules engines processes are spawned at a time. Workaround: The root cause is the splunk phased_execution_mode. Edit the limits.conf file and add the line:  [search] phased_execution_mode = auto
2021-01-21	ITSI-13167	On Safari, there is a 10 to 15 second delay when editing a Notable Event Aggregation Policy using the ServiceNow action

Notable Event Aggregation Policies

Date filed	Issue number	Description
2022-12-20	ITSI-27751	Episode Review arbitrary search filter with AND & OR conditions fail to match events under certain scenarios Workaround: Avoid using brackets (), extra whitespaces, the operator !=, and double quotes "" in the search filter
2022-01-25	ITSI-21269	The grouping of Bidirectional Ticketing events sets the episode KV store state to faulty values
2022-01-05	ITSI-20978	ServiceNow bidirectional integration resets the episode title, status, severity, and owner. Workaround: #In etc/app/itsi/local/savedsearches.conf, replace the Bidirectional Ticketing clause with the following: \[Bidirectional Ticketing] action.itsi_event_generator.param.description = %group_description% action.itsi_event_generator.param.itsi_instruction = %group_instruction% action.itsi_event_generator.param.owner = %group_assignee% action.itsi_event_generator.param.severity = %group_severity% action.itsi_event_generator.param.status = %group_status% action.itsi_event_generator.param.title = %group_title% disabled = 0 dispatch.earliest_time = -4h search = | `itsi_bidirectional_ticketing(main,90,itsi_tracked_alerts)`\\ | join group_id\\     \[ search `itsi_event_management_group_index_with_state("")` | rename itsi_group_id as group_id | stats latest(itsi_group_title) as group_title latest(severity) as group_severity latest(owner) as group_assignee latest(itsi_group_description) as group_description latest(itsi_group_instruction) as group_instruction latest(status) as group_status by group_id] If any NEAP action rules has the following condition Create SNOW ticket when number of events is equal to 1, there is a chance that the issue will occasionally come up due to a race condition between the initialization of the episode and the creation of the SNOW ticket. Change the activation criteria to when number of events is equal to 2 (or anything else) Increase action_execution_delay in itsi_rules_engine.properties to a higher value (i.e. 100) Restart Splunk.
2021-12-07	ITSI-20343	Impacted Services and KPIs do not appear in Episode Review when using Teams functionality Workaround: Create/edit Template:SA-ITOA/local/macros.conf and add the following two stanzas: {noformat}# Take in a string of team_keys in the format of '(sec_grp="itsi_team_key") OR (sec_grp="itsi_team_key")' and returns a filter of service_ids [itsi_events_compare_teams(1)] args = itsi_team_id_list definition = search (service_ids=*null*) OR (NOT service_ids=*) OR [|inputlookup itsi_services_in_team_lookup where ($itsi_team_id_list$) | rename _key as service_ids | eval service_ids="*".service_ids."*" | fields service_ids] Take in a string of team_keys in the format of '(sec_grp="itsi_team_key") OR (sec_grp="itsi_team_key")' and returns a filter of itsi_service_ids [itsi_groups_compare_teams(1)] args = itsi_team_id_list definition = search (itsi_service_ids=*null*) OR (NOT itsi_service_ids=*) OR [|inputlookup itsi_services_in_team_lookup where ($itsi_team_id_list$) | rename _key as itsi_service_ids | eval itsi_service_ids="*".itsi_service_ids."*" | fields itsi_service_ids]{noformat}
2021-12-03	ITSI-20314	Episode not being marked as inactive when bulk close is used
2021-10-20	ITSI-19415	On Windows server, more than 1 rules engines processes are spawned at a time. Workaround: The root cause is the splunk phased_execution_mode. Edit the limits.conf file and add the line:  [search] phased_execution_mode = auto
2021-01-21	ITSI-13167	On Safari, there is a 10 to 15 second delay when editing a Notable Event Aggregation Policy using the ServiceNow action

KPI Base Searches

Date filed	Issue number	Description
2022-05-24	ITSI-24346	KPI Reports Incorrect Values due to Auto-Generated Entity Filtering Logic Workaround: Removing the "extra" search_type= comparator before executing the KPI search again with the (edited) auto-generated entity filter returns expected results. Without doing so, not all expected fields and rows are always present

KPI Search Calculation

Date filed	Issue number	Description
2022-05-31	ITSI-24437	KPI with split by entity stops working after upgrade to 4.11.5. Workaround: This command seems to get the KPI calculation going again: /opt/splunk/bin/splunk cmd python /opt/splunk/etc/apps/SA-ITOA/bin/kvstore_to_json.py -m 4
2022-05-24	ITSI-24346	KPI Reports Incorrect Values due to Auto-Generated Entity Filtering Logic Workaround: Removing the "extra" search_type= comparator before executing the KPI search again with the (edited) auto-generated entity filter returns expected results. Without doing so, not all expected fields and rows are always present
2022-04-28	ITSI-23284	Deleted KPI lanes still showing in deep dive when the URL is refreshed.

Role Based Access Controls

Date filed	Issue number	Description
2022-12-19	ITSI-27734	different users with same role itoa_team_admin cannot modify saved SA Screen, works in 4.9.x and 4.13, broke in 4.11.x Workaround: Upgrade ITSI to version 4.13.x or 4.15.0

Service Analyzer

Date filed	Issue number	Description
2022-12-19	ITSI-27734	different users with same role itoa_team_admin cannot modify saved SA Screen, works in 4.9.x and 4.13, broke in 4.11.x Workaround: Upgrade ITSI to version 4.13.x or 4.15.0

Uncategorized issues

Date filed	Issue number	Description
2023-01-09	ITSI-27961	Bidirectional Ticketing Correlation Search hits "subsearch limit of 50000 reached" when the collection itsi_notable_event_ticketing has more than 50000 entries Workaround: # Navigate to ITSI -> Configuration -> Correlation Searches Click on Bidirectional Ticketing Paste the following search in the Search field and then click on Save. Also enable the CS if it has been disabled {noformat}| datamodel Ticket_Management Incident search | rename All_Ticket_Management.ticket_id as ticket_id | join ticket_id [search sourcetype="snow:incident" index="<snow_index>" | where _indextime > now() - <max_lookback_time>] | lookup itsi_notable_event_external_ticket tickets.ticket_id as ticket_id OUTPUTNEW tickets.ticket_system event_id | where isnotnull(event_id) | rename tickets.* as * | eventstats values(event_id) as group_id last(ticket_system) as ticket_system by ticket_id | fields - dv_* | table * | makemv group_id | mvexpand group_id | eval bidirectional_ticketing=1, snow_hash = number + "!" + group_id + "!" + sys_updated_on | search NOT [| search index="itsi_tracked_alerts" | fields snow_hash] | dedup snow_hash{noformat} Change the placeholders {{<snow_index>}} and {{<max_lookback_time>}} in the above search with values according to the customer's requirements
2022-03-24	ITSI-22641	Premium features disabled because the ITSI license checker is not finding all the valid licenses, when they are more than 30 licenses installed Workaround: If the customer has more than 30 licenses, remove the expired ones to keep the list short.
2022-02-16	ITSI-22140	Sorted by in Episode Review does not work with "First Event Time"
2022-02-04	ITSI-21526	Endpoint for closing and breaking an episode does not show a warning when all the fields are not provided in payload
2022-01-31	ITSI-21357	Critical issue if the Splunk Add-On for Windows and Windows forwarder (from ITSI Data Integration) are installed on same machine Workaround: To resolve the conflict, add disabled = 0 for all seven stanzas in input.conf file for the universal forwarder. You can find the universal forwarder input.conf file in this location: C:\\Program Files\\SplunkUniversalForwarder\\etc\\apps\\SplunkUniversalForwarder\\local\\input.conf. \[perfmon://CPU] \[perfmon://LogicalDisk] \[perfmon://Memory] \[perfmon://Network] \[perfmon://PhysicalDisk] \[perfmon://Process] \[perfmon://System]
2022-01-31	ITSI-21361	Bad regex in the ITSI Log Messages deduplicated panel of the ITSI health dashboard
2022-01-07	ITSI-21005	Discovery searches are not updating entities
2021-12-23	ITSI-20846	Bidirectional ticketing events being picked up as notable events and retitled as Ticket Event in episodes Workaround: # In etc/app/itsi/local/savedsearches.conf, replace the Bidirectional Ticketing clause with the following: {noformat}[Bidirectional Ticketing] action.itsi_event_generator.param.description = %group_description% action.itsi_event_generator.param.itsi_instruction = %group_instruction% action.itsi_event_generator.param.owner = %group_assignee% action.itsi_event_generator.param.severity = %group_severity% action.itsi_event_generator.param.status = %group_status% action.itsi_event_generator.param.title = %group_title% disabled = 0 dispatch.earliest_time = -4h search = | `itsi_bidirectional_ticketing(main,90,itsi_tracked_alerts)`\ | join group_id\ [ search `itsi_event_management_group_index_with_state("")` | rename itsi_group_id as group_id | stats latest(itsi_group_title) as group_title latest(itsi_group_severity) as group_severity latest(itsi_group_assignee) as group_assignee latest(itsi_group_description) as group_description latest(itsi_group_instruction) as group_instruction by group_id] {noformat} Restart Splunk This should help mitigate the issue while the code fix is in progress. If any of their NEAP action rules has the following condition Template:Create SNOW ticket when number of events is equal to 1, there is a chance that the issue will occasionally come up due to a race condition between the initialization of the episode and the creation of the SNOW ticket. In they notice this occurring, they can either: Change the activation criteria to Template:When number of events is equal to 2 (or anything else) Increase Template:Action execution delay in Template:Itsi rules engine.properties to a higher value (i.e. 100)
2021-12-14	ITSI-20605, ITSI-22366	Occasionally after ITSI upgrade, non-admin users get Oops Page - local.meta corrupted during the upgrade Workaround: Clean up all permissions on ITSI views in itsi/metadata/local.meta (and sync on SHC) The workaround is to clean up the stanza in local.meta on the all the SH. remove all the stanza like \[views/....]  that have no valid access settings, (access = delete : \[  ], read : \[  ], write : \[  ]) and that are not custom views from your users. As they may be many, to confirm, you can compare to the list in default.meta And you also can look at the modtime field in the stanza, as they are probably all identical.
2021-10-25	ITSI-19489	The Next Scheduled Time for entity management policies is based on the system time zone, instead of the user's current time zone.
2021-09-09	ITSI-18800	When you add ITSI instances as search peers to another Splunk instance, the peers might be disabled after 72 hours. This is because the ITSI licenses are flagged as duplicates on the search peers. Workaround: #Go to the node search peer manager node. Identify the Splunk licenses (Enterprise, ITSI, non-ITSI) currently installed. Ignore licenses under the *IT Service Intelligence Internals DO NOT COPY* stack. Navigate to http://LM_IP/en-US/manager/system/licensing/licenses and check if the AllowDuplicateKeys capability is enabled for each of the license identified in step 1. If not enabled, procure a new license from Splunk support and replace it. Make sure all licenses in the stack have the capability enabled. Restart Splunk.
2021-09-01	ITSI-18709	ITSI redirects to suite_redirect 500 Internal Server Error - because of python library isolation between apps Workaround: Step 1: Identify all the splunklib directories within the splunk apps directory using command find . -name 'splunklib' | xargs -r ls -lah. Step 2: For each directory listed in step 1, check if file six.py is present. Step 3: Copy the six.py from an existing splunklib directory into all the missing directories. Step 4: Clean the cached files using find . -name "*.pyc" -delete Step 5: Reload the ITE Work app.