Known issues in Splunk IT Service Intelligence

IT Service Intelligence (ITSI) version 4.11.4 has the following known issues and workarounds.

Backup/Restore and Migration Issues

Date filed	Issue number	Description
2021-10-13	ITSI-19215	Customer is getting a lot of errors related to "Could not find object id=itsi_entity_dashboard_drilldown" after installation of IT Essentials Work Workaround: Upgrade to ITE-Work version 4.12 and later

Bulk Import

Date filed	Issue number	Description
2023-04-12	ITSI-29489	module "ITSI Operating System" or other modules missing from entity import list options - when the SH has more than 30 DA-ITSI* apps installed
2021-06-09	ITSI-17178	Some ITSI Import Objects saved searches fail to merge entities with the host field and may create duplicate entities. Workaround: #Disable ITSI Import Objects - VMware VM. Copy the ITSI Import Objects - VMware VM saved search, but change the entity_merge_field attribute to host. Enable the updated ITSI Import Objects - VMware VM search.

Notable Events

Date filed	Issue number	Description
2022-12-20	ITSI-27751	Episode Review arbitrary search filter with AND & OR conditions fail to match events under certain scenarios Workaround: Avoid using brackets (), extra whitespaces, the operator !=, and double quotes "" in the search filter
2022-01-25	ITSI-21269	The grouping of Bidirectional Ticketing events sets the episode KV store state to faulty values
2022-01-21	ITSI-21232	Event not grouped when break_group_flag is set to false Workaround: Do not add the break_group_flag in the Notable Event if it's value is going to be false
2022-01-05	ITSI-20978	ServiceNow bidirectional integration resets the episode title, status, severity, and owner. Workaround: #In etc/app/itsi/local/savedsearches.conf, replace the Bidirectional Ticketing clause with the following: \[Bidirectional Ticketing] action.itsi_event_generator.param.description = %group_description% action.itsi_event_generator.param.itsi_instruction = %group_instruction% action.itsi_event_generator.param.owner = %group_assignee% action.itsi_event_generator.param.severity = %group_severity% action.itsi_event_generator.param.status = %group_status% action.itsi_event_generator.param.title = %group_title% disabled = 0 dispatch.earliest_time = -4h search = | `itsi_bidirectional_ticketing(main,90,itsi_tracked_alerts)`\\ | join group_id\\     \[ search `itsi_event_management_group_index_with_state("")` | rename itsi_group_id as group_id | stats latest(itsi_group_title) as group_title latest(severity) as group_severity latest(owner) as group_assignee latest(itsi_group_description) as group_description latest(itsi_group_instruction) as group_instruction latest(status) as group_status by group_id] If any NEAP action rules has the following condition Create SNOW ticket when number of events is equal to 1, there is a chance that the issue will occasionally come up due to a race condition between the initialization of the episode and the creation of the SNOW ticket. Change the activation criteria to when number of events is equal to 2 (or anything else) Increase action_execution_delay in itsi_rules_engine.properties to a higher value (i.e. 100) Restart Splunk.

Notable Event Aggregation Policies

Date filed	Issue number	Description
2022-12-20	ITSI-27751	Episode Review arbitrary search filter with AND & OR conditions fail to match events under certain scenarios Workaround: Avoid using brackets (), extra whitespaces, the operator !=, and double quotes "" in the search filter
2022-01-25	ITSI-21269	The grouping of Bidirectional Ticketing events sets the episode KV store state to faulty values
2022-01-21	ITSI-21232	Event not grouped when break_group_flag is set to false Workaround: Do not add the break_group_flag in the Notable Event if it's value is going to be false
2022-01-05	ITSI-20978	ServiceNow bidirectional integration resets the episode title, status, severity, and owner. Workaround: #In etc/app/itsi/local/savedsearches.conf, replace the Bidirectional Ticketing clause with the following: \[Bidirectional Ticketing] action.itsi_event_generator.param.description = %group_description% action.itsi_event_generator.param.itsi_instruction = %group_instruction% action.itsi_event_generator.param.owner = %group_assignee% action.itsi_event_generator.param.severity = %group_severity% action.itsi_event_generator.param.status = %group_status% action.itsi_event_generator.param.title = %group_title% disabled = 0 dispatch.earliest_time = -4h search = | `itsi_bidirectional_ticketing(main,90,itsi_tracked_alerts)`\\ | join group_id\\     \[ search `itsi_event_management_group_index_with_state("")` | rename itsi_group_id as group_id | stats latest(itsi_group_title) as group_title latest(severity) as group_severity latest(owner) as group_assignee latest(itsi_group_description) as group_description latest(itsi_group_instruction) as group_instruction latest(status) as group_status by group_id] If any NEAP action rules has the following condition Create SNOW ticket when number of events is equal to 1, there is a chance that the issue will occasionally come up due to a race condition between the initialization of the episode and the creation of the SNOW ticket. Change the activation criteria to when number of events is equal to 2 (or anything else) Increase action_execution_delay in itsi_rules_engine.properties to a higher value (i.e. 100) Restart Splunk.

Glass Table

Date filed	Issue number	Description
2022-03-25	ITSI-22654	Glass Table drilldown for ellipse widget does not work when data source is configured Workaround: For a better user experience, the minimum size of the visualization to support Drilldown is Charts: 60x60 SVs: 40x40 SVG, Image, Table, Punchcard: 10x10
2022-01-31	ITSI-21358	"Edit Permission" in Glass tables is not working when glass table is created using "create glass table" button Workaround: There are 2 workarounds User can change the permission of GT by checking the checkbox and selecting the bulk action → edit permissions. If user refreshes the page then user can change the permission of GT from edit → edit permissions.

KPI Search Calculation

Date filed	Issue number	Description
2023-05-17	ITSI-30211	Expensive KPI backfilll are causing indexers to crash, need to delete those Workaround: Make a Get call to get all backfill jobs matching service_tile matching "service_name" {noformat}curl -k -u <user> 'https://<host>:8089/servicesNS/nobody/SA-ITOA/storage/collections/data/itsi_backfill?query={%22service_title%22:%20/{%22$regex%22:%20%22service_name%22/}/}'{noformat} Run Delete call to clean up all of the backfill jobs matching the above criteria {noformat}curl -k -X DELETE -u <user> 'https://<host>:8089/servicesNS/nobody/SA-ITOA/storage/collections/data/itsi_backfill?query={%22service_title%22:%20/{%22$regex%22:%20%22service_name%22/}/}'{noformat}
2022-05-31	ITSI-24437	KPI with split by entity stops working after upgrade to 4.11.5. Workaround: This command seems to get the KPI calculation going again: /opt/splunk/bin/splunk cmd python /opt/splunk/etc/apps/SA-ITOA/bin/kvstore_to_json.py -m 4
2022-04-28	ITSI-23284	Deleted KPI lanes still showing in deep dive when the URL is refreshed.
2022-04-21	ITSI-23110	When summary index has huge data KPI edit workflow takes a long time from Step-1 to Step-2.
2022-01-10	ITSI-21013	With custom indexes, when creating new KPI, the backfill checks look to the default itsi_summary instead of the custom one, causing potentially extra backfill.

Role Based Access Controls

Date filed	Issue number	Description
2021-12-14	ITSI-20605, ITSI-22366	Occasionally after ITSI upgrade, non-admin users get Oops Page - local.meta corrupted during the upgrade Workaround: Clean up all permissions on ITSI views in itsi/metadata/local.meta (and sync on SHC) The workaround is to clean up the stanza in local.meta on the all the SH. remove all the stanza like \[views/....]  that have no valid access settings, (access = delete : \[  ], read : \[  ], write : \[  ]) and that are not custom views from your users. As they may be many, to confirm, you can compare to the list in default.meta And you also can look at the modtime field in the stanza, as they are probably all identical.

Service Analyzer

Date filed	Issue number	Description
2022-10-07	ITSI-26544	Service Analyzer returns no data because join_kpi_info macro's sub search hits the 50K limit
2022-02-17	ITSI-22146	Different users with same role itoa_team_admin cannot modify saved service analyzer. Workaround: Upgrade ITSI to version 4.13.x or 4.15.0

Service Health Score

Date filed	Issue number	Description
2022-09-28	ITSI-26376	Large number of KPI caused the service_health_metrics_monitor sub search to hit the 50000 default limit, causing discrepancies in values in Service Health Score alert_level in itsi_summary_metrics versus itsi_summary indexes. Workaround: Increase the limits.conf to adjust to the total number of KPIs in the subsearch of service_health_metrics_monitor. See example for a customer with 50000-70000 KPI objects. {{[join] }} Template:Subsearch maxout = 75000 {{#default was 50000 }} {{[searchresults] }} Template:Maxresultrows = 75000 {{ # default was 50000}}

Uncategorized issues

Date filed	Issue number	Description
2023-04-17	ITSI-29521	Fix the itsi_module_interface API to fetch more than 30 apps having prefix DA-ITSI
2023-01-09	ITSI-27961	Bidirectional Ticketing Correlation Search hits "subsearch limit of 50000 reached" when the collection itsi_notable_event_ticketing has more than 50000 entries Workaround: # Navigate to ITSI -> Configuration -> Correlation Searches Click on Bidirectional Ticketing Paste the following search in the Search field and then click on Save. Also enable the CS if it has been disabled {noformat}| datamodel Ticket_Management Incident search | rename All_Ticket_Management.ticket_id as ticket_id | join ticket_id [search sourcetype="snow:incident" index="<snow_index>" | where _indextime > now() - <max_lookback_time>] | lookup itsi_notable_event_external_ticket tickets.ticket_id as ticket_id OUTPUTNEW tickets.ticket_system event_id | where isnotnull(event_id) | rename tickets.* as * | eventstats values(event_id) as group_id last(ticket_system) as ticket_system by ticket_id | fields - dv_* | table * | makemv group_id | mvexpand group_id | eval bidirectional_ticketing=1, snow_hash = number + "!" + group_id + "!" + sys_updated_on | search NOT [| search index="itsi_tracked_alerts" | fields snow_hash] | dedup snow_hash{noformat} Change the placeholders {{<snow_index>}} and {{<max_lookback_time>}} in the above search with values according to the customer's requirements
2022-04-02	ITSI-22723	Upgr 4.9.6 >> 4.11.4 Failing
2022-03-24	ITSI-22641	Premium features disabled because the ITSI license checker is not finding all the valid licenses, when they are more than 30 licenses installed Workaround: If the customer has more than 30 licenses, remove the expired ones to keep the list short.
2022-03-03	ITSI-22366, ITSI-20605	Occasionally after upgrade to ITSI 4.9.*, non-admin users get Oops Page - local.meta corrupted during the upgrade - Dev Fix Work Workaround: Clean up all permissions on ITSI views in itsi/metadata/local.meta (and sync on SHC) The workaround is to clean up the stanza in local.meta on the all the SH. remove all the stanza like \[views/....]  that have no valid access settings, (access = delete : \[  ], read : \[  ], write : \[  ]) and that are not custom views from your users. As they may be many, to confirm, you can compare to the list in default.meta And you also can look at the modtime field in the stanza, as they are probably all identical.
2022-03-02	ITSI-22347	ITSI migration HTTP 500 - Sort operation used more than the maximum X bytes of RAM
2022-01-31	ITSI-21357	Critical issue if the Splunk Add-On for Windows and Windows forwarder (from ITSI Data Integration) are installed on same machine Workaround: To resolve the conflict, add disabled = 0 for all seven stanzas in input.conf file for the universal forwarder. You can find the universal forwarder input.conf file in this location: C:\\Program Files\\SplunkUniversalForwarder\\etc\\apps\\SplunkUniversalForwarder\\local\\input.conf. \[perfmon://CPU] \[perfmon://LogicalDisk] \[perfmon://Memory] \[perfmon://Network] \[perfmon://PhysicalDisk] \[perfmon://Process] \[perfmon://System]
2021-12-23	ITSI-20846	Bidirectional ticketing events being picked up as notable events and retitled as Ticket Event in episodes Workaround: # In etc/app/itsi/local/savedsearches.conf, replace the Bidirectional Ticketing clause with the following: {noformat}[Bidirectional Ticketing] action.itsi_event_generator.param.description = %group_description% action.itsi_event_generator.param.itsi_instruction = %group_instruction% action.itsi_event_generator.param.owner = %group_assignee% action.itsi_event_generator.param.severity = %group_severity% action.itsi_event_generator.param.status = %group_status% action.itsi_event_generator.param.title = %group_title% disabled = 0 dispatch.earliest_time = -4h search = | `itsi_bidirectional_ticketing(main,90,itsi_tracked_alerts)`\ | join group_id\ [ search `itsi_event_management_group_index_with_state("")` | rename itsi_group_id as group_id | stats latest(itsi_group_title) as group_title latest(itsi_group_severity) as group_severity latest(itsi_group_assignee) as group_assignee latest(itsi_group_description) as group_description latest(itsi_group_instruction) as group_instruction by group_id] {noformat} Restart Splunk This should help mitigate the issue while the code fix is in progress. If any of their NEAP action rules has the following condition Template:Create SNOW ticket when number of events is equal to 1, there is a chance that the issue will occasionally come up due to a race condition between the initialization of the episode and the creation of the SNOW ticket. In they notice this occurring, they can either: Change the activation criteria to Template:When number of events is equal to 2 (or anything else) Increase Template:Action execution delay in Template:Itsi rules engine.properties to a higher value (i.e. 100)
2021-09-01	ITSI-18709	ITSI redirects to suite_redirect 500 Internal Server Error - because of python library isolation between apps Workaround: Step 1: Identify all the splunklib directories within the splunk apps directory using command find . -name 'splunklib' | xargs -r ls -lah. Step 2: For each directory listed in step 1, check if file six.py is present. Step 3: Copy the six.py from an existing splunklib directory into all the missing directories. Step 4: Clean the cached files using find . -name "*.pyc" -delete Step 5: Restart Splunk on the ITE Work or ITSI search head.