Known issues in Splunk IT Service Intelligence

This version of IT Service Intelligence (ITSI) has the following known issues and workarounds.

Highlighted issues

Date filed	Issue number	Description
2023-05-06	ITSI-30026	Event generated from Provider are not getting grouped on Federated Search head Workaround: Event generated from provider gets grouped through the rule engine periodic backfill.

Adaptive Thresholding

Date filed	Issue number	Description
2023-07-23	ITSI-31404	Applyat command is less performant than itsiat command
2023-06-13	ITSI-30991	Outlier detection returns error StatisticsError: variance requires at least two data points for time series with valid data points

Backup/Restore and Migration Issues

Date filed	Issue number	Description
2024-01-09	ITSI-33724	Backup is not getting restored when custom saved searches are used in the service Workaround: * re-triggered the restore without any changes.
2023-10-12	ITSI-32459	Cleanup the migration_helper folder before the restore of the backup starts
2023-04-19	ITSI-29586	Unable to restore default scheduled backup Workaround: Download the Default Scheduled Backup and restore the downloaded backup

Deep Dive

Date filed	Issue number	Description
2023-08-09	ITSI-31640	Deep Dive overlay values are overlayed with the unit making it hard to read
2023-07-26	ITSI-31471	ITSI timepicker modifies the timerange for users that are using non default timezone

Entities

Date filed	Issue number	Description
2023-09-11	ITSI-32014	On Windows type entity_detail page, for the Process Monitoring Info table within Modal, after adjusting the column width, if we move the mouse, the height and width of the column changes Workaround: To resize a column, press Tab until the focus is on the column resize button handle, then use arrow left/right to resize.
2023-08-23	ITSI-31855, ITSI-33386	API entity_discovery_searches Failed to return discovery searches post upgrade Workaround: Once all the discovery searches related to the entity ran once, this issue will not exist. If the problematic search is 'disabled' and not intended to run anymore, can utilize the clean up command to clean this search out. ([1] ) If the problematic search simply has a run time that is much further in the future, then, you can change the cron schedule and let it run sooner and then change the time back. this way, you force the search to run again so the new status format gets saved.
2023-08-14	ITSI-31723	Error modal appears when user attempts to filter entities with a parenthesis in the name on entity management page Workaround: Use backslash before the special character. To search for "myhost(" try "myhost\("
2023-04-19	ITSI-29586	Unable to restore default scheduled backup Workaround: Download the Default Scheduled Backup and restore the downloaded backup

Notable Events

Date filed	Issue number	Description
2024-02-13	ITSI-34430	Groups Restore in rules engine should not be done based on the bidirectional ticketing events
2024-02-08	ITSI-34393	BDT event should not satisfy 'if episode is broken' action rule for inactive episodes
2023-11-26	ITSI-33166	Rules Engine process gets enabled after Splunk restart even if it is disabled Workaround: Enable High Scale EA Modular input under{{ Setting -> Data Inputs -> IT Service Intelligence High Scale Event Analytics Modular Input}}
2023-10-01	ITSI-32207	Previewing results in Notable Event Aggregation policies is not working for users without the "admin" or "scadmin" roles.
2023-09-21	ITSI-32156	preview results not working while NEAP creation in windows setup
2023-06-29	ITSI-31192	All Events tab does not render default columns if they are not present in NEAP JSON definition Workaround: # Use the latest ITSI Backup file to edit the NEAP JSON definition and remove the property Template:All events columns and restore the backup. Go to Episode Review page and add back all the desired columns
2023-06-19	ITSI-31057	host field value not visible to Rules Engine
2023-06-02	ITSI-30500	NEAP filtering criteria with value *(wildcard) does not satisfy the events which contain \n(line break) in the value Workaround: Add another negative filtering criteria for the field. For example, if we have added a filtering criteria Source Matches * then event will not be picked up by a custom NEAP. You need to add another negative filtering criteria with the or condition so it will be similar to Source Does not Match *.
2023-05-12	ITSI-30099	When multiple actions are triggered, field does not get updated according to the last action rule
2023-05-06	ITSI-30026	Event generated from Provider are not getting grouped on Federated Search head Workaround: Event generated from provider gets grouped through the rule engine periodic backfill.
2022-11-04	ITSI-27028	When Identifier Fields are specified for Notables and Smart Mode is enabled, the Episodes do not show the identifier fields

Notable Event Aggregation Policies

Date filed	Issue number	Description
2024-02-13	ITSI-34430	Groups Restore in rules engine should not be done based on the bidirectional ticketing events
2024-02-08	ITSI-34393	BDT event should not satisfy 'if episode is broken' action rule for inactive episodes
2023-11-26	ITSI-33166	Rules Engine process gets enabled after Splunk restart even if it is disabled Workaround: Enable High Scale EA Modular input under{{ Setting -> Data Inputs -> IT Service Intelligence High Scale Event Analytics Modular Input}}
2023-10-01	ITSI-32207	Previewing results in Notable Event Aggregation policies is not working for users without the "admin" or "scadmin" roles.
2023-09-21	ITSI-32156	preview results not working while NEAP creation in windows setup
2023-06-29	ITSI-31192	All Events tab does not render default columns if they are not present in NEAP JSON definition Workaround: # Use the latest ITSI Backup file to edit the NEAP JSON definition and remove the property Template:All events columns and restore the backup. Go to Episode Review page and add back all the desired columns
2023-06-19	ITSI-31057	host field value not visible to Rules Engine
2023-06-02	ITSI-30500	NEAP filtering criteria with value *(wildcard) does not satisfy the events which contain \n(line break) in the value Workaround: Add another negative filtering criteria for the field. For example, if we have added a filtering criteria Source Matches * then event will not be picked up by a custom NEAP. You need to add another negative filtering criteria with the or condition so it will be similar to Source Does not Match *.
2023-05-12	ITSI-30099	When multiple actions are triggered, field does not get updated according to the last action rule
2023-05-06	ITSI-30026	Event generated from Provider are not getting grouped on Federated Search head Workaround: Event generated from provider gets grouped through the rule engine periodic backfill.
2022-11-04	ITSI-27028	When Identifier Fields are specified for Notables and Smart Mode is enabled, the Episodes do not show the identifier fields

Glass Table

Date filed	Issue number	Description
2023-01-10	ITSI-27969	Ad hoc search should work properly even if we add it after deleting the existing the kpi data source from the visualization Workaround: Remove the value of options field from glass table source code in visualization when you delete the KPI data source and add adhoc data source in same visualization.
2023-01-05	ITSI-27888	Move Forward/Backward is not working when initially add visualization to a black canvas
2023-01-05	ITSI-27886	splunk.markdown adds unexpected background colour and text colour when leading spaces are used in text

KPI Base Searches

Date filed	Issue number	Description
2023-06-20	ITSI-31085	KPI Backfill searches run under 'Search' app context instead of ITSI/SA-ITOA app context
2023-02-23	ITSI-28869	Adhoc searches should not be validated while creating KPI base with metrics search option

KPI Search Calculation

Date filed	Issue number	Description
2023-09-13	ITSI-32031	itsi_at_search_kpi_minus7d Missing field alert_value at time
2023-06-20	ITSI-31085	KPI Backfill searches run under 'Search' app context instead of ITSI/SA-ITOA app context
2023-06-06	ITSI-30550	Show search/parser errors in UI for KPI creation model

Maintenance Window

Date filed	Issue number	Description
2023-10-18	ITSI-32628	Maintenance Window duration drop down is not updating End time value upon changing duration
2023-08-17	ITSI-31755	Incorrect default start time for the maintenance window
2023-07-17	ITSI-31345	Since upgrade ITSI to 4.17.0, filtering service at maintenance windows does not work.
2023-07-07	ITSI-31233	Changing the input time field in the maintenance window is not working as expected Workaround: # Open text editor Type time like 11:08:29.860 Copy the time Go to ITSI, click Text element to give focus Select all (select the current time) Paste
2023-06-30	ITSI-31195	maintenance window time setting issues

Performance

Date filed	Issue number	Description
2023-08-02	ITSI-31548	App SA-ITSI-AT-Recommendations failing Python3 readiness check Workaround: The customer is seeing a failing python upgrade readiness scan because we are coding exclusively in Python 3. We are not supporting Python 2 and as such, we were not previously using libraries that support Python 2 and Python 3 simultaneously (like Template:Six and Template:Future). Since Python for Scientific Computing is a requirement for Assisted Thresholding, we are already guaranteeing that the user will have Python 3 installed (our app gets Python from PSC, not from Splunk). For future reference, code from the Template:SA-ITSI-AT-Recommendations app does *not* need to pass the Python upgrade readiness scan

Role Based Access Controls

Date filed	Issue number	Description
2023-05-04	ITSI-30017	A user in itoa_user role cannot open ITSI homeview in SHC. Workaround: We have to add the list_search_head_clustering capability to the default authorize.conf.

Service Analyzer

Date filed	Issue number	Description
2023-10-02	ITSI-32214	Service analyzer link for service does not show up
2023-08-27	ITSI-31870	Sawtooth pattern for 15 min KPIs
2023-07-26	ITSI-31471	ITSI timepicker modifies the timerange for users that are using non default timezone
2023-06-21	ITSI-31097	In SA page filter service dropdown scroll is going in infinite loop Workaround: NA
2023-06-09	ITSI-30822	ITSI degraded-entities-search-manager may have caused indexers cluster to crash Workaround: If Service Analyser is running for more than 1 week's time range and search is going through millions of events try to limit the service analyser time range to less than 1 week to limit the search time range.
2023-06-07	ITSI-30580	When the dbconnect app is installed, non-admin ITSI users cannot access their homepage but are routed to the upgrade page. Workaround: Add the db_connect_read_app_conf capability to the custom user with a non-admin role. Enable this capability in the default authorize.conf file.
2023-05-10	ITSI-30080	Newly added entities in environment do not get linked to the services and their respective KPIs in ITSI v4.17.0 Workaround: Please re-install the Content Pack as a workaround for this issue. Follow below steps for the same: In ITSI, navigate to Configuration -> Data Integration Page Go to Content Library tab. Search and click on tile for Third-party APM Content Pack. Click on proceed button. Select objects from "Already Installed" tab in *Choose which objects to install* section. Click on "Replace Existing" radio button in *Choose a conflict resolution rule for the objects you install* section. If you want to enable the itsiObjects, then toggle the switch to enable the same in *Import as enabled (Optional) section*. (Optional) Click on install selected to install Content Pack. Wait for few minutes and then open Service Analyzer to view the data populated from ITSI objects of Third-party APM CP.

Service Definition

Date filed	Issue number	Description
2024-01-12	ITSI-33754	Simulated Health Score is not working as expected when service dependencies is add to service Workaround: After saving the service, Simulated Health Score calculation work as expected
2023-11-20	ITSI-33142	ITSI UI Bug - Service Template config page has button area above where button is displayed
2023-08-10	ITSI-31692	KPI cloning is not working when user select KPIs from more then one service.
2023-08-09	ITSI-31654	Should populate a list entities when you are trying to match entities in service definition
2023-08-02	ITSI-31556	Entity detail is missing data when viewing from Service Definition view
2023-05-29	ITSI-30378	Time policy auto-shifts by one hour upon user addition Workaround: Shifts time block by using kvstore_to_json.py script mode 3
2023-05-23	ITSI-30330	Improve performance of matched entities in entity rules tab

Service Health Score

Date filed	Issue number	Description
2024-01-12	ITSI-33754	Simulated Health Score is not working as expected when service dependencies is add to service Workaround: After saving the service, Simulated Health Score calculation work as expected
2023-07-14	ITSI-31329	Health score calculation - Importance level is not being saved

Service Templates

Date filed	Issue number	Description
2023-08-25	ITSI-31867	Entity rule fields are no longer in view only mode for the OOTB service templates

Predictive Analytics

Date filed	Issue number	Description
2023-07-26	ITSI-31471	ITSI timepicker modifies the timerange for users that are using non default timezone

Uncategorized issues

Date filed	Issue number	Description
2024-03-10	ITSI-34651	ITSI (Anomaly Detection) - Warning messages in "cohesive analysis" (Maximum entity limit is 30)
2024-02-29	ITSI-34551	Breaking event does not trigger breaking action rules
2024-01-23	ITSI-34041	ITSI Episode view triggers a search to populate linked tickets, that is looking back to epoch time=1 second till now
2024-01-12	ITSI-33757	stack "xerox" is missing "service sandbox" and "Custom Threshold Windows" from UI Workaround: If user has modified the navigation bar before the upgrade then it has created an entry in the Template:Local/data/ui/nav/default.xml and after the upgrade also Splunk will take the data from the local folder as it has the higher precedence even though it has the updated entry in the Template:Default/data/ui/nav/default.xml with new options. To fix this issue manually add the options from {{settings -> User interface -> Navigation menus}} and select default for the itsi app and add missing entry for the options from Template:Default/data/ui/nav/default.xml
2023-12-25	ITSI-33583	Episode review timeline should be updated when the policy filter is applied
2023-12-20	ITSI-33491	'index', 'splunk_server' and 'splunk_server_group' can not be added by 'Add Column' functionality on episode review page
2023-12-07	ITSI-33278	Cannot create a correlation search with all special character
2023-11-19	ITSI-33134	When the episode from 2nd page is selected and the table refreshed The focus from the episode is getting lost Workaround: The workaround for the issue mentioned here is, adding/updating the Template:Itsi notable group lookup macro from the Template:Etc/apps/SA-ITOA/local/macros.conf file. Add {{itsi_policy_id | eval policy_id=itsi_policy_id, _itsi_is_group_broken=if(is_active==0,1,0)}} at the end of the definition of the Template:Itsi notable group lookup macro so that it would look like below. {noformat}[itsi_notable_group_lookup] args = definition = lookup itsi_notable_group_user_lookup _key AS itsi_group_id OUTPUT owner severity status instruction | lookup itsi_notable_group_system_lookup _key AS itsi_group_id OUTPUT title description start_time last_time is_active event_count itsi_policy_id | eval policy_id=itsi_policy_id, _itsi_is_group_broken=if(is_active==0,1,0){noformat} This will make sure that the episode focus will not be lost for another 2 filters Template:Policy and Template:Show Episodes . So, for the filters Status, Owner, Severity, Policy and Show Episodes the episode focus will not be lost.
2023-11-15	ITSI-33113	Bulk Acknowledge Episode can be executed for already Acknowledged episode by another user
2023-11-09	ITSI-33057	The loadjob search is failing when adding "Event Fields" filter on episode review page
2023-11-08	ITSI-33041	itoa_user role cannot see all dashboards in dropdown in Windows Entity detail page
2023-10-19	ITSI-32660	Authored content packs does not preserve importance of ServiceHealthScore of dependent KPI
2023-10-19	ITSI-32657	Events not being indexed into itsi_tracked_alerts if SSL in not enabled Workaround: Go to Data Inputs -> HTTP Event Collector -> "Enable SSL" checkbox → Enable It
2023-10-17	ITSI-32621	Notable drildown search not working on Episode view with sourcetype and colon
2023-10-13	ITSI-32474, ITSI-34680	Adding Ticketing filter after editing columns in Episode Review and saving does not respect the saved state Workaround: Reload the Episode Reviev page after the save to reload the Event management state
2023-10-09	ITSI-32409	Policy filter does not showed up in the Episode Review eventhough user has access to policy details view
2023-10-01	ITSI-32208	The "itoa_interface/service" endpoint is not returning any results
2023-09-27	ITSI-32180, ITSI-33045	Installation Completed Modal for content packs does not display any warning/error message when no objects are installed
2023-09-18	ITSI-32093, ITSI-31750	In ITSI 4.17, when a user has no quota left to run a concurrent search to populate, the search now fails instead of queuing. This leads to errors in the Service Analyzer. Workaround: Increase search quotas (and dispatch size) for the role. Update the concurrent search quota by updating the settings srchJobsQuota and possibly rtSrchJobsQuota in authorize.conf for the appropriate roles.
2023-09-14	ITSI-32050	Unable to import entities (csv)- Import job fails (works for admin)
2023-09-08	ITSI-32000, ITSI-33662, ITSI-32413	When multiple episodes are linked to the same Ticket ID(ServiceNow), the filter dropdown for the "Tickets" option displays duplicate Ticket ID
2023-09-06	ITSI-31981	Unnamed deep dives are showing up in content pack authorship
2023-09-05	ITSI-31978	Correlation search edit page malfunctions when time range set to "All Time" Workaround: *Workaround 1* From Setting-->Searches, Reports, and Alerts-->search for CS-->Edit Alert-->Update time range to Template:Last 15 Minutes from Template:All Time Workaround 2* Update/add itsi/local/savedsearches.conf as following: {noformat}dispatch.earliest_time = 0 dispatch.latest_time = now{noformat}
2023-09-01	ITSI-31904	In upgrade scenario, the "Entity Discovery Searches" feature does not list the discovery search identifying entity.
2023-08-18	ITSI-31763	Closing events not excluded in periodic backfill search
2023-08-17	ITSI-31759	Special character in entity_status_tracking field will crash entity search page Workaround: Make sure no special characters are added to the Template:Entity status tracking field
2023-08-16	ITSI-31748	Unable to edit cron expression field for the ELM policy
2023-08-11	ITSI-31708	KPI backfilling is not working properly for services linked with more number of entities
2023-08-10	ITSI-31706	Drill down is not working as expected in XML dashboard
2023-08-09	ITSI-31657	No events in itsi_tracked_alerts and itsi_grouped alerts because of Invalid token Workaround: Disable and enabled the itsi_tracked_alerts token: Settings → Data inputs → HTTP Event Collector → Disabled and Enable the token whose index is itsi_tracked_alerts In the scenario where I saw this error message, itsi_grouped_alerts token was also impacted. If itsi_tracked_alerts issue is fixed and you are still not seeing the events in itsi_grouped_alerts, then enable and disable all the tokens related to ITSI EA. If disabling/enabling the token does not work, need to delete the tokens. For OnPrem, need to restart the SH for tokens to be repopulated, for Cloud it should be automatic.
2023-08-04	ITSI-31560	After upgrade ITSI to 4.17.0, episode no longer indexed into itsi_grouped_alerts Workaround: The issue is that there are conflicting entries present for the tokens under different apps. The token entries present in the Template:000-self-service app should be deleted from the apps/000-self-service/local/inputs.conf Template:Disabled=0  should be added for tokens which do not have this field in apps/SA-ITOA/local/inputs.conf
2023-08-02	ITSI-31555, ITSI-31464	the ITSI integration create SNOW tickets with SPL instead of INC prefix when using Episode Action with custom endpoints with ServiceNow_TA version 7.6.0 Workaround: Until bug in service now ADDON 7.6 bug (ADDON-64098 & ADDON-63502 ) are resolved, to avoid the issue, in ITSI, do not specify a custom endpoint in the action setup, keep the field empty.
2023-07-17	ITSI-31339	Service templates has references of services that are not selected in the custom Content Pack
2023-06-13	ITSI-30946	Upgrade to ITSI 4.17.0 fails when the management port is changed from the default value of 8089
2023-05-24	ITSI-30342	Restoring recurring and currently applicable custom threshold window is not getting activated
2023-05-17	ITSI-30231, ITSI-30289	change entity lookup field to old format
2023-05-11	ITSI-30097, ITSI-32375	Status chart on entity type health page should display all the appropriate statuses
2023-05-10	ITSI-30068	Event Analytics Monitoring Rules Engine Information panel uses an All time search Workaround: Edit the Event Analytics Monitoring Dashboard. Click on the magnifying glass under the Rules Engine Information panel. Change the Time Range to Shared Time Picker (time_token).
2023-04-20	ITSI-29608	itsi_bidirectional_ticking macro should use macros for index and source types being used within the search
2023-04-20	ITSI-29609, ITSI-30886	itsi_bmc_bidirectional_ticking macro should use macros for index and source types being used within the search
2022-09-19	ITSI-26219, ITSI-26290	Support of splunk.choropleth.svg is missing in ITSI partial backup, Content authorship and itsimodels
2021-08-22	ITSI-18480	ITSI license is checker unable to parse descriptions with multiple lines in server.conf Workaround: Check all the licenses and the license-pools in the license-manager, and find the one that contains custom Template:Description or notes. If they contains linebreaks, remove the linebreaks from the notes.