Known issues in Splunk IT Service Intelligence

IT Service Intelligence (ITSI) version 4.3.1 has the following known issues and workarounds.

Splunk platform issues that impact ITSI compatibility

Date filed	Issue number	Description
2019-02-14	SPL-155648	ITSI event analytics is incompatible with Splunk Enterprise version 7.2.0 - 7.2.3. On versions 7.1.x and 7.2.4 - 7.2.10, event analytics might duplicate events. To work around these issues, create a limits.conf file on all search heads at $SPLUNK_HOME/etc/apps/SA-ITOA/local/ and add the following stanza: [search] phased_execution_mode = auto If you do not plan on using event analytics, the above does not apply. See Splunk Enterprise system requirement in the Install and Upgrade Splunk IT Service Intelligence manual.

Adaptive Thresholding

Date filed	Issue number	Description
2019-12-07	ITSI-5035	Adaptive thresholds occasionally don't update at midnight.

Backup/Restore and Migration Issues

Date filed	Issue number	Description
2019-11-20	ITSI-4917	During backup/restore, notable event archiving fails for large KV store collections, causing the restore to be very slow. Workaround: Check whether ITSI shows the following error message in the internal logs at source=itsi_notable_event_archive-age_notable_event.log: 2019-12-06 10:39:06,951 ERROR [itsi.notable_event_archive] [__init__] [exception] [15399] [HTTP 500] Splunkd internal error; [{'type': 'ERROR', 'code': None, 'text': 'An error occurred. (Internal read failed with error code \'96\' and message \'Executor error during OP_QUERY find :: caused by :: errmsg: "Sort operation used more than the maximum 33554432 bytes of RAM. Add an index, or specify a smaller limit."\')'}] If so, add acceleration on the large collections to reduce the memory pressure during a sort. Open $SPLUNK_HOME/etc/apps/SA-ITOA/local/collections.conf and add the following stanzas for the impacted collections: #if the collection group_system is too large [itsi_notable_group_system] accelerated_fields.mod_time = {"mod_time": 1} # if the group_user is too large [itsi_notable_group_user] accelerated_fields.mod_time = {"mod_time": 1}
2019-07-24	ITSI-3836	Objects such as service analyzers, glass tables, and deep dives are missing after upgrade. Workaround: If some objects are missing from the UI or unaccessible after you upgrade, the ACL objects corresponding to the objects might be missing or corrupted. For troubleshooting steps, see https://docs.splunk.com/Documentation/ITSI/latest/Install/Troubleshoot.
2019-07-23	ITSI-3830	Upon upgrade to 4.3.x, correlation searches and multi-KPI alerts are missing the 'all_info' field.
2019-06-11	ITSI-3452	Upon upgrade in a search head cluster, the event grouping custom command "itsirulesengine" may fail to run on some search heads: "ERROR Unable to invoke factory method in class class org.apache.logging.log4j.core.config.PropertiesPlugin". Workaround: To validate the root cause, log in to each search head and run the following search: | itsirulesengine If the search fails on a search head, an error message appears in the UI and in the search.log. Once you have identified the offending search head, perform one of the following actions: 1. SSH to the search head and remove the following files: cd /opt/splunk/etc/apps/SA-ITOA/lib/java/event_management/libs rm akka-actor_2.11-2.3.15.jar akka-slf4j_2.11-2.3.15.jar config-1.2.1.jar log4j-api-2.3.jar log4j-core-2.3.jar log4j-slf4j-impl-2.3.jar scala-library-2.11.5.jar slf4j-api-1.7.21.jar Then retry the search. 2. If the files were pushed from the deployer, go to the deployer and remove the files: cd /opt/splunk/shcluster/apps/SA-ITOA/lib/java/event_management/libs rm akka-actor_2.11-2.3.15.jar akka-slf4j_2.11-2.3.15.jar config-1.2.1.jar log4j-api-2.3.jar log4j-core-2.3.jar log4j-slf4j-impl-2.3.jar scala-library-2.11.5.jar slf4j-api-1.7.21.jar Then push the bundle to the search head and retry the search.
2019-05-07	ITSI-3119	Upgrade fails because a service template sync was queued. Workaround: Delete the backup using the curl command to change its status to Completed. Then force the service template sync. Restart Splunk software to complete the migration.
2019-01-03	ITSI-2164	ITSI backup times out due to an extremely large number of episode comments in the KV store. Workaround: Delete all comments prior to the backup (purge the collections in the KV store) or increase the Splunkd timeout and KV store limits. Then reduce the lifetime of the ITSI notable event collections in the KV store to archive them faster (the default is 6 months).
2018-10-16	ITSI-1748	You cannot restore an ITSI backup more than once. Workaround: This issue occurs because the saved search DA-ITSI-APM-EUEM_Base_Search is missing from the system. Create the missing saved search manually before restoring the backup. For example, create a local version of savedsearches.conf and add the following stanza: [DA-ITSI-APM-EUEM_Base_Search] description = search = request.ui_dispatch_app = itsi request.ui_dispatch_view = search
2017-02-10	ITSI-1309	If multiple services use one KPI base search, and the total size of your services exceeds 50 MB, ITSI generates an error. Workaround: Increase the value for max_size_per_batch_save_mb (50MB is default) in $SPLUNK_HOME/etc/apps/SA-ITOA/local/limits.conf under the [kvstore] stanza.
2016-05-02	ITSI-1305	After migration, shared objects (service analyzers, glass tables, and deep dives) are not accessible. Workaround: Use the curl command and create ACLs for each of the shared objects that are currently saved in the KV store collections: itsi_pages and itsi_service_analyzer. For example: $ curl -u admin:Splunk3r -k https://127.0.0.1:8089/servicesNS/nobody/SA-UserAccess/storage/collections/data/app_acl -X POST -H "Content-Type:application/json" -d '\{ "obj_id": "XXX-XXX-XXX", "obj_type": "glass_table", "obj_app": "itsi", "obj_storename": "itsi_pages", "obj_acl": \{ "obj_owner": "nobody", "read": ["*"], "write": ["*"], "delete": ["*"] }, "object_shared_by_inclusion": "true", "acl_owner": "nobody" }'

Bulk Import

Date filed	Issue number	Description
2019-09-17	ITSI-4402	Scheduled entity import isn't importing new entities. Workaround: Open $SPLUNK_HOME/etc/apps/SA-ITOA/local/inputs.conf. In the itsi_csv_import stanza, make sure the entity_merge_field setting is set to a blank value. Remove the word undefined if it's there.
2019-07-10	ITSI-3723	The modular input for recurring entity import says "undefined" for the entity_merge_field. Workaround: 1. Open the local copy of inputs.conf at $SPLUNK_HOME/etc/apps/SA-ITOA/local. 2. Locate the itsi_csv_import stanza for the recurring entity import. 3. Instead of entity_merge_field = undefined, change the setting to entity_merge_field = (an empty string).
2018-12-20	ITSI-2147	When importing entities from CSV, if entities have conflicting identifier and informational fields, a syntax error occurs and the page hangs. Workaround: The import still works in the backend. Ignore the hanging page and check the logs to make sure the import was successful. Additionally, a Splunk message should pop up and say that bulk import has completed successfully.
2015-03-25	ITSI-1293	In a search head cluster environment, you cannot set up a recurring import (from CSV or search) through the UI. Workaround: 1. Create the modular input through the UI. ITSI adds the input as a new stanza in $SPLUNK_HOME/etc/apps/itsi/local/inputs.conf. It is not replicated across search peers. Alternatively, if you're familiar with the format of modular inputs, you can create the input yourself. 2.Copy the input stanza from the local version of inputs.conf and add it to shcluster/apps/itsi/local/inputs.conf on the deployer. 3. Let the deployer push the file to the search peers. The file is deployed to the default inputs.conf on each search peer. 4. Remove the modular input stanza from $SPLUNK_HOME/etc/apps/itsi/local/inputs.conf on the search head that created it. Otherwise it will take precedence on the deployer.

Deep Dive

Date filed	Issue number	Description
2019-05-22	ITSI-3258	"HTTP 414: URI Too Long" when navigating in the ITSI UI. Workaround: ITSI does not limit URL length, so pages with too many characters fail to load. To work around this issue, limit your request lengths to the following: Browser request: < 2048 characters REST request: < 8192 characters.
2016-12-14	ITSI-525	If you zoom in on a specific time range in a deep dive while using twin-lane comparison, the comparisons that appear are occasionally offset by up to a minute.

Entities

Date filed	Issue number	Description
2019-10-28	ITSI-4721	Commas in the value of an entity's alias create duplicate entity aliases.
2015-02-12	ITSI-1286	When importing entities using Data inputs > IT Service Intelligence CSV Import, the page overflows.

Notable Events

Date filed	Issue number	Description
2020-06-28	ITSI-9183	For time-based aggregation policies, new events are added to a broken episode after the Rules Engine restarts.
2020-05-19	ITSI-8412	Acknowledged episodes get unassigned by themselves when the episode break period elapses. Workaround: No workaround is available.
2020-05-05	ITSI-7536	Time-based retention policies don't work properly for the itsi_notable_group_user KV store collection because the mod_time field is missing.
2020-05-05	ITSI-7535	You can't run the notable event retention policy on collections with more than 200K objects.
2020-05-05	ITSI-7537	When archiving more than 50K objects in a KV store collection at a time, only the first 50K are archived in the index, while the remaining objects are removed from the KV store.
2020-02-27	ITSI-5932	ITSI doesn't support running Splunk Enterprise version 8.0.x with Ubuntu 18.04 and Open JDK 11. Workaround: Use Oracle JDK 11 or Open/Oracle JDK 8 instead of Open JDK 11, or use other versions of Linux.
2020-01-16	ITSI-5402	Rules Engine GC Collection Memory Exhaustion
2019-11-20	ITSI-4940	Nothing blocks you from creating an external ticket from an episode for which a ticket was already created.
2019-10-15	ITSI-4663	Upon upgrade to version 4.3.0 or later, the Rules Engine command fails with error: "Error occurred during initialization of VM". Workaround: This issue occurs because 32-bit Java cannot run the Rules Engine with the new memory settings introduced in version 4.3.x. Open or create a local copy of commands.conf at $SPLUNK_HOME/etc/apps/SA-ITOA/local. Add the following stanza: [itsirulesengine] command.arg.1=-J-Xmx1024M # reduced to 1024MB for 32 bit JDK/JRE Restart the Rules Engine, either by disabling and reenabling the itsi_event_grouping search, or by restarting Splunk software.
2019-10-09	ITSI-4606	ITSI backups keep failing and notable event KV store collections are growing very large. Workaround: This issue occurs because the indexed realtime search returns events over and over from buckets that use tsidx reduction. Disable tsidx reduction on the itsi_tracked_alerts and itsi_summary indexes and rebuild all old buckets on these indexes.
2019-08-21	ITSI-4149	Smart Mode continues adding notable events to episodes even after the you manually close the episodes.
2019-06-13	ITSI-3483, ITSI-3382	When using the "Link Ticket" option in Episode Review, the URL redirects to the wrong page. Workaround: Make sure the URL starts with http:// or https://. Otherwise the URL is interpreted as a relative URI.
2019-06-11	ITSI-3452	Upon upgrade in a search head cluster, the event grouping custom command "itsirulesengine" may fail to run on some search heads: "ERROR Unable to invoke factory method in class class org.apache.logging.log4j.core.config.PropertiesPlugin". Workaround: To validate the root cause, log in to each search head and run the following search: | itsirulesengine If the search fails on a search head, an error message appears in the UI and in the search.log. Once you have identified the offending search head, perform one of the following actions: 1. SSH to the search head and remove the following files: cd /opt/splunk/etc/apps/SA-ITOA/lib/java/event_management/libs rm akka-actor_2.11-2.3.15.jar akka-slf4j_2.11-2.3.15.jar config-1.2.1.jar log4j-api-2.3.jar log4j-core-2.3.jar log4j-slf4j-impl-2.3.jar scala-library-2.11.5.jar slf4j-api-1.7.21.jar Then retry the search. 2. If the files were pushed from the deployer, go to the deployer and remove the files: cd /opt/splunk/shcluster/apps/SA-ITOA/lib/java/event_management/libs rm akka-actor_2.11-2.3.15.jar akka-slf4j_2.11-2.3.15.jar config-1.2.1.jar log4j-api-2.3.jar log4j-core-2.3.jar log4j-slf4j-impl-2.3.jar scala-library-2.11.5.jar slf4j-api-1.7.21.jar Then push the bundle to the search head and retry the search.
2019-04-09	ITSI-2916	Episode Review displays "NaN" values in the event count column. Workaround: Either refresh the browser or refresh the Episode Review dashboard.
2019-02-15	ITSI-2532	Notable event aggregation policies occasionally don't pass tokens to actions.
2019-01-03	ITSI-2164	ITSI backup times out due to an extremely large number of episode comments in the KV store. Workaround: Delete all comments prior to the backup (purge the collections in the KV store) or increase the Splunkd timeout and KV store limits. Then reduce the lifetime of the ITSI notable event collections in the KV store to archive them faster (the default is 6 months).
2018-12-10	ITSI-2059	Some notable events are added to more than one episode. Workaround: For an ITSI search head running Splunk 7.1 or 7.2, create or edit etc/system/local/limits.conf and add the following stanza:  [search] phased_execution_mode = auto   For an ITSI search head running Splunk 7.3 or later, there is no need to change anything.
2017-03-29	ITSI-1316	Splunkd connection fails due to "no_shared cipher matched" between client and server. Workaround: In order for notable event management and anomaly detection to work with Splunk platform 6.6, do the following: Java 8/JRE 1.8/JDK 1.8* * Download JCE 8 from http://www.oracle.com/technetwork/java/javase/downloads/jce8-download-2133166.html * Unzip the downloaded file * Place the two jars from the zip file into <java_jre_install_dir>/lib/security/ if running the JRE or <java_jdk_install_dir>/jre/lib/security if running the JDK. Java 7/JRE 1.7/JDK 1.7* * Download JCE 7 from http://www.oracle.com/technetwork/java/javase/downloads/jce-7-download-432124.html * Unzip the downloaded file * Place the two jars from the zip file into <java_jre_install_dir>/lib/security/ if running the JRE or <java_jdk_install_dir>/jre/lib/security if running the JDK. Update SA-ITOA/local/commands.conf with the following commands:  [itsirulesengine] type = custom command.arg.1=-J-Xmx1024M command.arg.2=-Dlog4j.configurationFile=../default/log4j_rules_engine.xml command.arg.3=-DitsiRulesEngine.configurationFile=../default/itsi_rules_engine.properties command.arg.4=-Dhttps.protocols=TLSv1.2,TLSv1.1 command.arg.5=-Dhttps.cipherSuites=TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256,TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384,TLS_RSA_WITH_AES_128_CBC_SHA256 chunked = true [itsicorrelationengine] type = custom command.arg.1=-J-Xmx1024M command.arg.2=-Dlog4j.configurationFile=../default/log4j_correlation_engine.xml command.arg.3=-J-XX:+UseConcMarkSweepGC command.arg.4=-DitsiCorrelationEngine.configurationFile=../default/itsi_correlation_engine.properties command.arg.5=-Dhttps.protocols=TLSv1.2,TLSv1.1 command.arg.6=-Dhttps.cipherSuites=TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256,TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384,TLS_RSA_WITH_AES_128_CBC_SHA256 chunked = true Update SA-ITSI-MetricAD/local/commands.conf with the following commands:    [mad] type = custom command.arg.1=-J-Xmx1G command.arg.2=-Dlog4j.configurationFile=../default/log4j.xml command.arg.3=-Dlog4j2.threadContextMap=com.splunk.mad.util.MadThreadContextMapcommand.arg.4=-Dhttps.protocols=TLSv1.2,TLSv1.1 command.arg.5=-Dhttps.cipherSuites=TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256,TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384,TLS_RSA_WITH_AES_128_CBC_SHA256chunked = true
2017-03-29	ITSI-1299	When your browser and the Splunk server are set to different DST time zones, the incorrect time might display for events in Episode Review. Workaround: Set your time zone to something other than "system default" even if you are in the same time zone as the system default.
2016-09-08	ITSI-1268	ITSI generates duplicate event_ids from the itsi_tracked_alerts index. This occurs when correlation search results contain an existing event_id. In this case, ITSI picks up the value of the event_id field and does not create a GUID for the event. Workaround: Rename the event_id field.
2016-04-01	ITSI-1346	The 'Ping Host' action does not work when ITSI and Enterprise Security are installed on the same machine. Workaround: 1. Add the following stanza to $SPLUNK_HOME/etc/apps/SplunkEnterpriseSecurity/local/inputs.conf: [app_imports_update://update_es] apps_to_update = (SA-(?!(ITOA|ITSI|IndexCreation|UserAccess)).*) | (Splunk_SA_.*) 2. Delete the "import = *" line from [] stanza of $SPLUNK_HOME/etc/apps/$APP/metadata/local.meta, where APP=SA-ITOA, SA-ITSI-ATAD, SA-ITSI-LicenseChecker, SA-IndexCreation, SA-UserAccess. 3. Restart Splunk.

Notable Event Aggregation Policies

Date filed	Issue number	Description
2020-06-28	ITSI-9183	For time-based aggregation policies, new events are added to a broken episode after the Rules Engine restarts.
2020-05-19	ITSI-8412	Acknowledged episodes get unassigned by themselves when the episode break period elapses. Workaround: No workaround is available.
2020-05-05	ITSI-7536	Time-based retention policies don't work properly for the itsi_notable_group_user KV store collection because the mod_time field is missing.
2020-05-05	ITSI-7535	You can't run the notable event retention policy on collections with more than 200K objects.
2020-05-05	ITSI-7537	When archiving more than 50K objects in a KV store collection at a time, only the first 50K are archived in the index, while the remaining objects are removed from the KV store.
2020-02-27	ITSI-5932	ITSI doesn't support running Splunk Enterprise version 8.0.x with Ubuntu 18.04 and Open JDK 11. Workaround: Use Oracle JDK 11 or Open/Oracle JDK 8 instead of Open JDK 11, or use other versions of Linux.
2020-01-16	ITSI-5402	Rules Engine GC Collection Memory Exhaustion
2019-11-20	ITSI-4940	Nothing blocks you from creating an external ticket from an episode for which a ticket was already created.
2019-10-15	ITSI-4663	Upon upgrade to version 4.3.0 or later, the Rules Engine command fails with error: "Error occurred during initialization of VM". Workaround: This issue occurs because 32-bit Java cannot run the Rules Engine with the new memory settings introduced in version 4.3.x. Open or create a local copy of commands.conf at $SPLUNK_HOME/etc/apps/SA-ITOA/local. Add the following stanza: [itsirulesengine] command.arg.1=-J-Xmx1024M # reduced to 1024MB for 32 bit JDK/JRE Restart the Rules Engine, either by disabling and reenabling the itsi_event_grouping search, or by restarting Splunk software.
2019-10-09	ITSI-4606	ITSI backups keep failing and notable event KV store collections are growing very large. Workaround: This issue occurs because the indexed realtime search returns events over and over from buckets that use tsidx reduction. Disable tsidx reduction on the itsi_tracked_alerts and itsi_summary indexes and rebuild all old buckets on these indexes.
2019-08-21	ITSI-4149	Smart Mode continues adding notable events to episodes even after the you manually close the episodes.
2019-06-13	ITSI-3483, ITSI-3382	When using the "Link Ticket" option in Episode Review, the URL redirects to the wrong page. Workaround: Make sure the URL starts with http:// or https://. Otherwise the URL is interpreted as a relative URI.
2019-06-11	ITSI-3452	Upon upgrade in a search head cluster, the event grouping custom command "itsirulesengine" may fail to run on some search heads: "ERROR Unable to invoke factory method in class class org.apache.logging.log4j.core.config.PropertiesPlugin". Workaround: To validate the root cause, log in to each search head and run the following search: | itsirulesengine If the search fails on a search head, an error message appears in the UI and in the search.log. Once you have identified the offending search head, perform one of the following actions: 1. SSH to the search head and remove the following files: cd /opt/splunk/etc/apps/SA-ITOA/lib/java/event_management/libs rm akka-actor_2.11-2.3.15.jar akka-slf4j_2.11-2.3.15.jar config-1.2.1.jar log4j-api-2.3.jar log4j-core-2.3.jar log4j-slf4j-impl-2.3.jar scala-library-2.11.5.jar slf4j-api-1.7.21.jar Then retry the search. 2. If the files were pushed from the deployer, go to the deployer and remove the files: cd /opt/splunk/shcluster/apps/SA-ITOA/lib/java/event_management/libs rm akka-actor_2.11-2.3.15.jar akka-slf4j_2.11-2.3.15.jar config-1.2.1.jar log4j-api-2.3.jar log4j-core-2.3.jar log4j-slf4j-impl-2.3.jar scala-library-2.11.5.jar slf4j-api-1.7.21.jar Then push the bundle to the search head and retry the search.
2019-04-09	ITSI-2916	Episode Review displays "NaN" values in the event count column. Workaround: Either refresh the browser or refresh the Episode Review dashboard.
2019-02-15	ITSI-2532	Notable event aggregation policies occasionally don't pass tokens to actions.
2019-01-03	ITSI-2164	ITSI backup times out due to an extremely large number of episode comments in the KV store. Workaround: Delete all comments prior to the backup (purge the collections in the KV store) or increase the Splunkd timeout and KV store limits. Then reduce the lifetime of the ITSI notable event collections in the KV store to archive them faster (the default is 6 months).
2018-12-10	ITSI-2059	Some notable events are added to more than one episode. Workaround: For an ITSI search head running Splunk 7.1 or 7.2, create or edit etc/system/local/limits.conf and add the following stanza:  [search] phased_execution_mode = auto   For an ITSI search head running Splunk 7.3 or later, there is no need to change anything.
2017-03-29	ITSI-1316	Splunkd connection fails due to "no_shared cipher matched" between client and server. Workaround: In order for notable event management and anomaly detection to work with Splunk platform 6.6, do the following: Java 8/JRE 1.8/JDK 1.8* * Download JCE 8 from http://www.oracle.com/technetwork/java/javase/downloads/jce8-download-2133166.html * Unzip the downloaded file * Place the two jars from the zip file into <java_jre_install_dir>/lib/security/ if running the JRE or <java_jdk_install_dir>/jre/lib/security if running the JDK. Java 7/JRE 1.7/JDK 1.7* * Download JCE 7 from http://www.oracle.com/technetwork/java/javase/downloads/jce-7-download-432124.html * Unzip the downloaded file * Place the two jars from the zip file into <java_jre_install_dir>/lib/security/ if running the JRE or <java_jdk_install_dir>/jre/lib/security if running the JDK. Update SA-ITOA/local/commands.conf with the following commands:  [itsirulesengine] type = custom command.arg.1=-J-Xmx1024M command.arg.2=-Dlog4j.configurationFile=../default/log4j_rules_engine.xml command.arg.3=-DitsiRulesEngine.configurationFile=../default/itsi_rules_engine.properties command.arg.4=-Dhttps.protocols=TLSv1.2,TLSv1.1 command.arg.5=-Dhttps.cipherSuites=TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256,TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384,TLS_RSA_WITH_AES_128_CBC_SHA256 chunked = true [itsicorrelationengine] type = custom command.arg.1=-J-Xmx1024M command.arg.2=-Dlog4j.configurationFile=../default/log4j_correlation_engine.xml command.arg.3=-J-XX:+UseConcMarkSweepGC command.arg.4=-DitsiCorrelationEngine.configurationFile=../default/itsi_correlation_engine.properties command.arg.5=-Dhttps.protocols=TLSv1.2,TLSv1.1 command.arg.6=-Dhttps.cipherSuites=TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256,TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384,TLS_RSA_WITH_AES_128_CBC_SHA256 chunked = true Update SA-ITSI-MetricAD/local/commands.conf with the following commands:    [mad] type = custom command.arg.1=-J-Xmx1G command.arg.2=-Dlog4j.configurationFile=../default/log4j.xml command.arg.3=-Dlog4j2.threadContextMap=com.splunk.mad.util.MadThreadContextMapcommand.arg.4=-Dhttps.protocols=TLSv1.2,TLSv1.1 command.arg.5=-Dhttps.cipherSuites=TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256,TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384,TLS_RSA_WITH_AES_128_CBC_SHA256chunked = true
2017-03-29	ITSI-1299	When your browser and the Splunk server are set to different DST time zones, the incorrect time might display for events in Episode Review. Workaround: Set your time zone to something other than "system default" even if you are in the same time zone as the system default.
2016-09-08	ITSI-1268	ITSI generates duplicate event_ids from the itsi_tracked_alerts index. This occurs when correlation search results contain an existing event_id. In this case, ITSI picks up the value of the event_id field and does not create a GUID for the event. Workaround: Rename the event_id field.
2016-04-01	ITSI-1346	The 'Ping Host' action does not work when ITSI and Enterprise Security are installed on the same machine. Workaround: 1. Add the following stanza to $SPLUNK_HOME/etc/apps/SplunkEnterpriseSecurity/local/inputs.conf: [app_imports_update://update_es] apps_to_update = (SA-(?!(ITOA|ITSI|IndexCreation|UserAccess)).*) | (Splunk_SA_.*) 2. Delete the "import = *" line from [] stanza of $SPLUNK_HOME/etc/apps/$APP/metadata/local.meta, where APP=SA-ITOA, SA-ITSI-ATAD, SA-ITSI-LicenseChecker, SA-IndexCreation, SA-UserAccess. 3. Restart Splunk.

Glass Table

Date filed	Issue number	Description
2020-02-12	ITSI-5749	When you upgrade a glass table to the beta framework, KPI icons lose their color.
2019-10-09	ITSI-4617	KPI and ad hoc searches in beta glass tables are run under the context "App: Search" instead of "App: ITSI". This prevents certain app-restricted searches such as Predictive Analytics searches from running.
2019-09-11	ITSI-4351	The beta glass table framework doesn't back up background images.
2019-07-08	ITSI-3680	You can't edit the title or description of existing beta glass tables from the glass table lister page. Workaround: Reload the GT lister page.
2019-06-17	ITSI-3505, SCP-13983	Adding a drilldown from a Column or Area visualization causes infinite redirection to the drilldown link.
2018-09-14	ITSI-1567	When you add a predictive model to a glass table, you cannot use the sparkline or trending value viz types because the prediction is a static value.

KPI Base Searches

Date filed	Issue number	Description
2020-01-07	ITSI-5220	Shared base searches are generating thousands of "Broken Socket" messages but ITSI functionality is not impacted.
2019-09-06	ITSI-4275	Multiple KPIs that are linked to shared base searches fail to populate. Workaround: Increase timeouts in ITSI commands. Add the following stanza to $SPLUNK_HOME/etc/apps/SA-ITOA/local/itsi_settings.conf: [customsearch] timeout_read = 360 Add the following setting to $SPLUNK_HOME/etc/apps/SA-ITOA/lib/ITOA/storage/statestore.py: REST_TIMEOUT = 6000 You must reimplement this workaround the next time you upgrade ITSI.

Maintenance Window

Date filed	Issue number	Description
2020-10-07	ITSI-11354	The maintenance window UI calculation of the daylight saving starting day is incorrect. Workaround: Check the start time displayed as a preview (in UTC) when creating maintenance windows to ensure that your maintenance window is created correctly.

Role Based Access Controls

Date filed	Issue number	Description
2019-05-15	ITSI-3203	Episode Review always displays "You may not have permissions to view some events in this episode" when a role is missing certain capabilities. Workaround: When a user has a role that is lacking the capabilities "delete_isti_team" and "write_itsi_team", Episode Review cannot evaluate if notable events are restricted to specific team. The message is systematic. If you're using the Global team, the message is irrelevant.
2019-03-29	ITSI-2860	If you assign the write_itsi_correlation_search capability to the itoa_analyst role, the role still cannot create a correlation search. Workaround: In addition to assigning the write_itsi_correlation_search capability to the itoa_analyst role, create a local.meta file at SPLUNK_HOME/etc/apps/itsi/metadata/ and add "itoa_analyst" to the [savedsearches] stanza. For example: [savedsearches] access = read : [ * ], write: [ itoa_admin, itoa_team_admin, itoa_analyst ], delete: [ itoa_admin, itoa_team_admin, itoa_analyst ] export = system

Service Analyzer

Date filed	Issue number	Description
2019-11-12	ITSI-4826	Entity rules are overriden when imported services linked to a service template are split due to the import_batch_size.
2019-05-22	ITSI-3258	"HTTP 414: URI Too Long" when navigating in the ITSI UI. Workaround: ITSI does not limit URL length, so pages with too many characters fail to load. To work around this issue, limit your request lengths to the following: Browser request: < 2048 characters REST request: < 8192 characters.
2017-10-04	ITSI-1290	Filters with no matching results can't be saved in the Service Analyzer.

Service Definition

Date filed	Issue number	Description
2016-03-28	ITSI-1269	On Windows 10 on Chrome, some selectors in the ITSI app do not function.

Threshold Templates

Date filed	Issue number	Description
2019-04-08	ITSI-2914	When you first add a new KPI to a service template and apply Adaptive Thresholding, the additional KPI reuses the preview of the first KPI that was added to the template and displays misleading threshold values. Workaround: Once the scheduled daily adaptive threshold update runs, all KPIs linked to the template are correctly updated. Wait until midnight for the adaptive threshold values to update themselves.

Predictive Analytics

Date filed	Issue number	Description
2019-10-09	ITSI-4617	KPI and ad hoc searches in beta glass tables are run under the context "App: Search" instead of "App: ITSI". This prevents certain app-restricted searches such as Predictive Analytics searches from running.
2019-10-01	ITSI-4530, ITSI-4604	The KPI Predictions chart on the Predictive Analytics dashboard does not display the correct timestamps.
2019-10-01	ITSI-4531	The Predictive Analytics Dashboard "KPI Predictions" panel plots results in GMT rather than the user's timezone.
2019-03-20	ITSI-2801	Predictive Analytics occasionally fails to train models on Windows. Workaround: If search.log for the fit command reports the following error: ERROR ChunkedExternProcessor - stderr: ImportError: DLL load failed: The application has failed to start because its side-by-side configuration is incorrect. Please see the application event log or use the command-line sxstrace.exe tool for more detail. To resolve this issue, reinstall of Visual C++ 2008 runtime: [1]
2018-09-14	ITSI-1567	When you add a predictive model to a glass table, you cannot use the sparkline or trending value viz types because the prediction is a static value.
2018-08-01	ITSI-1105	After you delete a Predictive Analytics model through Lookups, the model still appears in the UI.

Splunk App for Infrastructure Integration

Date filed	Issue number	Description
2019-05-21	ITSI-3248	The itoa_admin role does not have permission to create alerts in SAI.
2018-09-24	ITSI-1654	Only 50,000 entities can be imported from the Splunk App for Infrastructure. Workaround: By default, the entity integration imports up to 50,000 entities from the Splunk App for Infrastructure. If you have more than 50,000 entities in Splunk App for Infrastructure, only the first 50,000 will be imported into ITSI. Increase the max_rows_per_query setting in $SPLUNK_HOME/etc/apps/SA-ITOA/local/limits.conf under the [kvstore] stanza to import more than 50,000 entities.

Uncategorized issues

Date filed	Issue number	Description
2020-02-04	ITSI-5615	Token based authentication failing to authenticate on Search-Head-Cluster for SA-ITOA REST API interfaces Workaround: When calling rest endpoint, for example: curl --insecure -H "Authorization: Bearer -X GET  <TOKEN>" https://localhost:8089/servicesNS/nobody/SA-ITOA/itoa_interface/v4.3.1/service/ You will see messages like this: {"message":"(500, '[HTTP 401] Client is not authenticated')"}. This issue is caused by SPL-183142. Only affects SH clusters but not standalone instance, from 7.3 onwards. Specifically impacts apps that use rest endpoints. The workaround is to use the old way of authenticating, username/password.
2020-01-29	ITSI-5553	When you restore any service from a partial backup, the restore changes the thresholds of other services.
2019-12-11	ITSI-5055	When you import services through a service template, the KPIs are recreated, causing issues with anomaly detection and backfill.
2019-08-27	ITSI-4190	The itoa_team_admin role can't create a service or entity via import from CSV or search.
2019-08-23	ITSI-4171	When your system's time zone and the Splunk time zone set in your user preferences are different, it may cause several hours of lag between Rules Engine logs and Python logs in the _internal index. Workaround: Configure your Splunk time zone to be the same as your system's time zone.
2019-07-31	ITSI-3902	There are excessive InsecureRequestWarning messages in splunkd.log when using Python 2 libraries. Workaround: Migrate ITSI and Splunk Enterprise to Python 3. For instructions, see https://docs.splunk.com/Documentation/ITSI/latest/Install/Python3.
2019-07-01	ITSI-3666	Upon upgrade, the Splunk product name changes from Splunk>enterprise to Splunk>hunk. Workaround: Ensure you have active group defined in server.conf [license] active_group = Enterprise
2019-06-08	ITSI-3437	Correlation searches don't work with real-time searches.
2019-05-30	ITSI-3322	If you add a correlation search in ITSI which contains a sub-search returning into an eval, you get a message "Invalid search string: This search cannot be parsed when parse_only is set to true." Workaround: You can't use a sub-search returning into an eval in a correlation search. As a workaround, create and save a basic correlation search with all of the information you want outside of the search. Then as an admin user, go to Settings > Searches, reports, and alerts and open the correlation search you just created. Add the sub-search you were trying to add there.
2019-02-19	ITSI-2550	Using "/" in role names causes issues with REST calls initiated by ITSI. Workaround: Change role names to omit / characters
2019-02-12	ITSI-2471	If ITSI is installed on multiple environments with multiple license masters, and any indexer interacts with both environments, a duplicate licensing error occurs because both environments have the same auto-generated ITSI license stack. Workaround: Follow the workaround described in the deployment planning docs for the version of ITSI you're currently using: https://docs.splunk.com/Documentation/ITSI/latest/Install/Plan#ITSI_license_requirements
2018-06-27	ITSI-1287, ITSI-793	Correlation searches created by manually editing savedsearches.conf do not appear on the correlation search lister page. Workaround: Do not create correlation searches by manually editing $SPLUNK_HOME/etc/apps/itsi/local/savedsearches.conf. The search will not appear on the correlation search lister page. Always create correlation searches directly in the IT Service Intelligence app.
2015-12-01	ITSI-1320	When you install Enterprise Security on a search head with a pre-existing installation of ITSI, the ES-specific roles overwrite the ITSI-specific roles assigned to admin role. This disables access to all read/write objects in ITSI. Workaround: 1. In Splunk Web, go to Settings > Access Controls. 2. Select Roles > admin. 3. Add itoa_admin, itoa_analyst, and itoa_user to Selected roles. 4. Click Save.

All ITSI Modules

Publication date	Issue number	Description
2017-03-21	ITOA-7585	When you bulk add services and an error caused by the racing condition occurs, the incorrect message "itsi_module does not exist" is displayed.
2017-03-07	MOD-979	KPIs do not have consistent backfill settings across all modules.
2017-01-17	MOD-452	The Analyze KPI button on the Service Details page is broken.
2017-01-17	MOD-402	The Export to PDF option does not work in the drilldown to a module.
2017-01-17	MOD-296	The extendable tab XML generator REST endpoint is located in DA-ITSI-OS instead of in common components where it can be used by all modules.
2017-01-17	MOD-591	ITSI displays a misleading error message when a KPI template contains a field that cannot be resolved.
2017-01-17	MOD-498	There is no upper limit to the number of characters a KPI title or description can contain. Long strings can negatively affect performance.
2017-01-17	MOD-309	The Gruntfile.js included in ITSI modules uses double quotes instead of single quotes, which does not conform to the standard for all JavaScript files.
2017-04-17	MOD-2002	When you drilldown from the Events tab, an "Invalid earliest_time" error occurs. Workaround: Disable drilldown from the Events tab.
2017-01-17	MOD-439	Some modules do not have descriptions for saved searches.

Application Server Module

Publication date	Issue number	Description
2017-01-27	MOD-492	If you reuse the same panel within a dashboard, the duplicate panel does not display any event data.

Cloud Services Module

There are no known issues for this release.

Database Module

Publication date	Issue number	Description
2017-01-17	MOD-586	When a lookup is not configured for TA-Microsoft-SqlServer, ITSI displays a misleading error message on the server drilldown page.

End User Experience Module

There are no known issues for this release.

Load Balancer Module

Publication date	Issue number	Description
2017-01-27	MOD-492	If you reuse the same panel within a dashboard, the duplicate panel does not display any event data.

Operating System Module

Publication date	Issue number	Description
2017-04-13	MOD-555	The Storage Free Space % base search runs every minute while the Linux df command runs every 5 minutes. This causes data gaps.
2017-04-10	MOD-1964	Windows data for memory free space is collected at different intervals than the Memory Free % KPI.
2017-01-17	MOD-1398	Line, stack, and area charts do not display a metric gap when no metrics are available during a time period.

Storage Module

There are no known issues for this release.

Virtualization Module

There are no known issues for this release.

Web Server Module

Publication date	Issue number	Description
2017-03-17	MOD-320	Some KPI ad hoc searches transform data with the stats command and do not retain time fields. The KPIs do not render anything and do not show thresholding details.
2017-03-17	MOD-538	When you add a new tab with panels and refresh the page, the page breaks.