Known issues in Splunk IT Service Intelligence

IT Service Intelligence (ITSI) version 4.11.5 has the following known issues and workarounds.

Backup/Restore and Migration Issues

Date filed	Issue number	Description
2022-09-15	ITSI-26204	ITSI Default Scheduled Backup taking hours to complete after upgrade to 4.11.5 (it used to be minutes) Workaround: * Run the below curl command to delete the entry in the collection Template:Itsi migration status {noformat}curl -ku admin https://localhost:8089/servicesNS/nobody/SA-ITOA/storage/collections/data/itsi_migration_status -X DELETE{noformat}

Bulk Import

Date filed	Issue number	Description
2021-06-09	ITSI-17178	Some ITSI Import Objects saved searches fail to merge entities with the host field and may create duplicate entities. Workaround: #Disable ITSI Import Objects - VMware VM. Copy the ITSI Import Objects - VMware VM saved search, but change the entity_merge_field attribute to host. Enable the updated ITSI Import Objects - VMware VM search.

Deep Dive

Date filed	Issue number	Description
2022-05-19	ITSI-24186	Auto save for a default deep dive is not working.

Entities

Date filed	Issue number	Description
2022-01-18	ITSI-21193	Splunk dashboard's input dropdown searches aren't running within ITE Work entity detail view Workaround: The following dashboards are affected by this issue: Applications Crashes Application Installs Event Monitoring Network Activity Windows Update To work around this issue, add earliest and latest values to dashboard XML. Follow these steps to add the earliest and latest values to the XML: Go to Dashboards > Dashboards. Select the dashboard from the list. Select Edit. Select Source. Add earliest and latest values inside the <search> tags for all dropdowns: Select Save. Example dashboard XML: <search> <query>| inputlookup windows_netmon_system | dedup Host | sort Host</query> <earliest>0</earliest> <latest>now</latest> </search> See https://docs.splunk.com/Documentation/ITSI/4.12.0/Entity/EntityType#configure-time-range-picker-tokens-in-your-dashboards for more info.

Notable Events

Date filed	Issue number	Description
2022-12-20	ITSI-27751	Episode Review arbitrary search filter with AND & OR conditions fail to match events under certain scenarios Workaround: Avoid using brackets (), extra whitespaces, the operator !=, and double quotes "" in the search filter
2022-07-06	ITSI-24871	NEAP breaking criteria not obeying OR condition when time based conditions are selected Workaround: Keep the time based breaking conditions i.e. "_If this episode existed for:_ " and "_if the flow of events in the episode paused for:_" at the end of the OR conditions and the "_The following event occurs_" condition as first.
2022-06-30	ITSI-24808	ITSI rules of episode breaking conditions don't work Workaround: Keep the time based breaking conditions i.e. "_If this episode existed for:_ " and "_if the flow of events in the episode paused for:_" at the end of the OR conditions and the "_The following event occurs_" condition as first.

Notable Event Aggregation Policies

Date filed	Issue number	Description
2022-12-20	ITSI-27751	Episode Review arbitrary search filter with AND & OR conditions fail to match events under certain scenarios Workaround: Avoid using brackets (), extra whitespaces, the operator !=, and double quotes "" in the search filter
2022-07-06	ITSI-24871	NEAP breaking criteria not obeying OR condition when time based conditions are selected Workaround: Keep the time based breaking conditions i.e. "_If this episode existed for:_ " and "_if the flow of events in the episode paused for:_" at the end of the OR conditions and the "_The following event occurs_" condition as first.
2022-06-30	ITSI-24808	ITSI rules of episode breaking conditions don't work Workaround: Keep the time based breaking conditions i.e. "_If this episode existed for:_ " and "_if the flow of events in the episode paused for:_" at the end of the OR conditions and the "_The following event occurs_" condition as first.

KPI Search Calculation

Date filed	Issue number	Description
2022-05-31	ITSI-24437	KPI with split by entity stops working after upgrade to 4.11.5. Workaround: This command seems to get the KPI calculation going again: /opt/splunk/bin/splunk cmd python /opt/splunk/etc/apps/SA-ITOA/bin/kvstore_to_json.py -m 4
2022-04-28	ITSI-23284	Deleted KPI lanes still showing in deep dive when the URL is refreshed.
2022-04-21	ITSI-23110	When summary index has huge data KPI edit workflow takes a long time from Step-1 to Step-2.
2022-01-10	ITSI-21013	With custom indexes, when creating new KPI, the backfill checks look to the default itsi_summary instead of the custom one, causing potentially extra backfill.

Role Based Access Controls

Date filed	Issue number	Description
2021-12-14	ITSI-20605, ITSI-22366	Occasionally after ITSI upgrade, non-admin users get Oops Page - local.meta corrupted during the upgrade Workaround: Clean up all permissions on ITSI views in itsi/metadata/local.meta (and sync on SHC) The workaround is to clean up the stanza in local.meta on the all the SH. remove all the stanza like \[views/....]  that have no valid access settings, (access = delete : \[  ], read : \[  ], write : \[  ]) and that are not custom views from your users. As they may be many, to confirm, you can compare to the list in default.meta And you also can look at the modtime field in the stanza, as they are probably all identical.

Service Analyzer

Date filed	Issue number	Description
2023-02-17	ITSI-28826	Changes to health score color values in threshold_labels.conf do not appear in the service analyzer.

Uncategorized issues

Date filed	Issue number	Description
2023-01-09	ITSI-27961	Bidirectional Ticketing Correlation Search hits "subsearch limit of 50000 reached" when the collection itsi_notable_event_ticketing has more than 50000 entries Workaround: # Navigate to ITSI -> Configuration -> Correlation Searches Click on Bidirectional Ticketing Paste the following search in the Search field and then click on Save. Also enable the CS if it has been disabled {noformat}| datamodel Ticket_Management Incident search | rename All_Ticket_Management.ticket_id as ticket_id | join ticket_id [search sourcetype="snow:incident" index="<snow_index>" | where _indextime > now() - <max_lookback_time>] | lookup itsi_notable_event_external_ticket tickets.ticket_id as ticket_id OUTPUTNEW tickets.ticket_system event_id | where isnotnull(event_id) | rename tickets.* as * | eventstats values(event_id) as group_id last(ticket_system) as ticket_system by ticket_id | fields - dv_* | table * | makemv group_id | mvexpand group_id | eval bidirectional_ticketing=1, snow_hash = number + "!" + group_id + "!" + sys_updated_on | search NOT [| search index="itsi_tracked_alerts" | fields snow_hash] | dedup snow_hash{noformat} Change the placeholders {{<snow_index>}} and {{<max_lookback_time>}} in the above search with values according to the customer's requirements
2022-08-22	ITSI-25886	Fix BDT event back filling due to not indexed into grouped alert
2022-08-09	ITSI-25749	Vital metrics data doesn't populate when there are more than 100 entities in ITSI
2022-07-13	ITSI-24985	Entities don't retain metadata on becoming inactive for conflict resolution of type replace.
2022-07-11	ITSI-24902	ITSI entity management functionality flags previously detected entities as unstable after upgrade to newer versions. Workaround: Delete the existing entities and let the existing entities be re-discovered during next run of the discovery search.
2022-03-24	ITSI-22641	Premium features disabled because the ITSI license checker is not finding all the valid licenses, when they are more than 30 licenses installed Workaround: If the customer has more than 30 licenses, remove the expired ones to keep the list short.
2022-01-31	ITSI-21357	Critical issue if the Splunk Add-On for Windows and Windows forwarder (from ITSI Data Integration) are installed on same machine Workaround: To resolve the conflict, add disabled = 0 for all seven stanzas in input.conf file for the universal forwarder. You can find the universal forwarder input.conf file in this location: C:\\Program Files\\SplunkUniversalForwarder\\etc\\apps\\SplunkUniversalForwarder\\local\\input.conf. \[perfmon://CPU] \[perfmon://LogicalDisk] \[perfmon://Memory] \[perfmon://Network] \[perfmon://PhysicalDisk] \[perfmon://Process] \[perfmon://System]
2021-09-01	ITSI-18709	ITSI redirects to suite_redirect 500 Internal Server Error - because of python library isolation between apps Workaround: Step 1: Identify all the splunklib directories within the splunk apps directory using command find . -name 'splunklib' | xargs -r ls -lah. Step 2: For each directory listed in step 1, check if file six.py is present. Step 3: Copy the six.py from an existing splunklib directory into all the missing directories. Step 4: Clean the cached files using find . -name "*.pyc" -delete Step 5: Restart Splunk on the ITE Work or ITSI search head.