
Known issues in Splunk IT Service Intelligence
IT Service Intelligence (ITSI) version 4.3.0 has the following known issues and workarounds.
Splunk platform issues that impact ITSI compatibility
Date filed | Issue number | Description |
---|---|---|
2019-02-14 | SPL-155648 |
[search] phased_execution_mode = auto
See Splunk Enterprise system requirement in the Install and Upgrade Splunk IT Service Intelligence manual. |
Backup/Restore and Migration Issues
Date filed | Issue number | Description |
---|---|---|
2019-06-11 | ITSI-3452 | Upon upgrade to 4.2.x in a search head cluster, the event grouping custom command "itsirulesengine" may fail to run on some search heads: "ERROR Unable to invoke factory method in class class org.apache.logging.log4j.core.config.PropertiesPlugin". Workaround: To validate the root cause, log in to each search head and run the following search:
cd /opt/splunk/etc/apps/SA-ITOA/lib/java/event_management/libs rm akka-actor_2.11-2.3.15.jar akka-slf4j_2.11-2.3.15.jar config-1.2.1.jar log4j-api-2.3.jar log4j-core-2.3.jar log4j-slf4j-impl-2.3.jar scala-library-2.11.5.jar slf4j-api-1.7.21.jar Then retry the search. cd /opt/splunk/shcluster/apps/SA-ITOA/lib/java/event_management/libs rm akka-actor_2.11-2.3.15.jar akka-slf4j_2.11-2.3.15.jar config-1.2.1.jar log4j-api-2.3.jar log4j-core-2.3.jar log4j-slf4j-impl-2.3.jar scala-library-2.11.5.jar slf4j-api-1.7.21.jar Then push the bundle to the search head and retry the search. |
2019-05-07 | ITSI-3119 | Upgrade fails because a service template sync was queued. Workaround: Delete the backup using the curl command to change its status to Completed. Then force the service template sync. Restart Splunk software to complete the migration. |
2019-01-03 | ITSI-2164 | ITSI backup times out due to an extremely large number of episode comments in the KV store. Workaround: Delete all comments prior to the backup (purge the collections in the KV store) or increase the Splunkd timeout and KV store limits. Then reduce the lifetime of the ITSI notable event collections in the KV store to archive them faster (the default is 6 months). |
2018-10-16 | ITSI-1748 | You cannot restore an ITSI backup more than once. Workaround: This issue occurs because the saved search DA-ITSI-APM-EUEM_Base_Search is missing from the system. Create the missing saved search manually before restoring the backup. For example, create a local version of savedsearches.conf and add the following stanza: [DA-ITSI-APM-EUEM_Base_Search] description = search = request.ui_dispatch_app = itsi request.ui_dispatch_view = search |
2017-02-10 | ITSI-1309 | If multiple services use one KPI base search, and the total size of your services exceeds 50 MB, ITSI generates an error. Workaround: Increase the value for max_size_per_batch_save_mb (50MB is default) in $SPLUNK_HOME/etc/apps/SA-ITOA/local/limits.conf under the [kvstore] stanza. |
2016-05-02 | ITSI-1305 | After migration, shared objects (service analyzers, glass tables, and deep dives) are not accessible. Workaround: Use the curl command and create ACLs for each of the shared objects that are currently saved in the KV store collections: itsi_pages and itsi_service_analyzer. For example: $ curl -u admin:Splunk3r -k https://127.0.0.1:8089/servicesNS/nobody/SA-UserAccess/storage/collections/data/app_acl -X POST -H "Content-Type:application/json" -d '\{ "obj_id": "XXX-XXX-XXX", "obj_type": "glass_table", "obj_app": "itsi", "obj_storename": "itsi_pages", "obj_acl": \{ "obj_owner": "nobody", "read": ["*"], "write": ["*"], "delete": ["*"] }, "object_shared_by_inclusion": "true", "acl_owner": "nobody" }' |
Deep Dive
Date filed | Issue number | Description |
---|---|---|
2019-05-22 | ITSI-3258 | "HTTP 414: URI Too Long" when navigating in the ITSI UI. Workaround: ITSI does not limit URL length, so pages with too many characters fail to load. To work around this issue, limit your request lengths to the following:
|
2016-12-14 | ITSI-525 | If you zoom in on a specific time range in a deep dive while using twin-lane comparison, the comparisons that appear are occasionally offset by up to a minute. |
Entities
Date filed | Issue number | Description |
---|---|---|
2015-02-12 | ITSI-1286 | When importing entities using Data inputs > IT Service Intelligence CSV Import, the page overflows. |
Notable Events
Date filed | Issue number | Description |
---|---|---|
2019-06-13 | ITSI-3483, ITSI-3382 | When using the "Link Ticket" option in Episode Review, the URL redirects to the wrong page. Workaround: Make sure the URL starts with http:// or https:// . Otherwise the URL is interpreted as a relative URI. |
2019-06-11 | ITSI-3452 | Upon upgrade to 4.2.x in a search head cluster, the event grouping custom command "itsirulesengine" may fail to run on some search heads: "ERROR Unable to invoke factory method in class class org.apache.logging.log4j.core.config.PropertiesPlugin". Workaround: To validate the root cause, log in to each search head and run the following search:
cd /opt/splunk/etc/apps/SA-ITOA/lib/java/event_management/libs rm akka-actor_2.11-2.3.15.jar akka-slf4j_2.11-2.3.15.jar config-1.2.1.jar log4j-api-2.3.jar log4j-core-2.3.jar log4j-slf4j-impl-2.3.jar scala-library-2.11.5.jar slf4j-api-1.7.21.jar Then retry the search. cd /opt/splunk/shcluster/apps/SA-ITOA/lib/java/event_management/libs rm akka-actor_2.11-2.3.15.jar akka-slf4j_2.11-2.3.15.jar config-1.2.1.jar log4j-api-2.3.jar log4j-core-2.3.jar log4j-slf4j-impl-2.3.jar scala-library-2.11.5.jar slf4j-api-1.7.21.jar Then push the bundle to the search head and retry the search. |
2019-01-03 | ITSI-2164 | ITSI backup times out due to an extremely large number of episode comments in the KV store. Workaround: Delete all comments prior to the backup (purge the collections in the KV store) or increase the Splunkd timeout and KV store limits. Then reduce the lifetime of the ITSI notable event collections in the KV store to archive them faster (the default is 6 months). |
2018-12-10 | ITSI-2059 | Some notable events are added to more than one episode. Workaround: For an ITSI search head running Splunk 7.1 or 7.2, create or edit etc/system/local/limits.conf and add the following stanza: [search] phased_execution_mode = auto For an ITSI search head running Splunk 7.3 or later, there is no need to change anything. |
2017-03-29 | ITSI-1299 | When your browser and the Splunk server are set to different DST time zones, the incorrect time might display for events in Episode Review. Workaround: Set your time zone to something other than "system default" even if you are in the same time zone as the system default. |
2017-03-29 | ITSI-1316 | Splunkd connection fails due to "no_shared cipher matched" between client and server. Workaround: In order for notable event management and anomaly detection to work with Splunk platform 6.6, do the following:
* Download JCE 8 from http://www.oracle.com/technetwork/java/javase/downloads/jce8-download-2133166.html * Unzip the downloaded file * Place the two jars from the zip file into <java_jre_install_dir>/lib/security/ if running the JRE or <java_jdk_install_dir>/jre/lib/security if running the JDK.
* Download JCE 7 from http://www.oracle.com/technetwork/java/javase/downloads/jce-7-download-432124.html * Unzip the downloaded file * Place the two jars from the zip file into <java_jre_install_dir>/lib/security/ if running the JRE or <java_jdk_install_dir>/jre/lib/security if running the JDK. Update SA-ITOA/local/commands.conf with the following commands: [itsirulesengine] type = custom command.arg.1=-J-Xmx1024M command.arg.2=-Dlog4j.configurationFile=../default/log4j_rules_engine.xml command.arg.3=-DitsiRulesEngine.configurationFile=../default/itsi_rules_engine.properties command.arg.4=-Dhttps.protocols=TLSv1.2,TLSv1.1 command.arg.5=-Dhttps.cipherSuites=TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256,TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384,TLS_RSA_WITH_AES_128_CBC_SHA256 chunked = true [itsicorrelationengine] type = custom command.arg.1=-J-Xmx1024M command.arg.2=-Dlog4j.configurationFile=../default/log4j_correlation_engine.xml command.arg.3=-J-XX:+UseConcMarkSweepGC command.arg.4=-DitsiCorrelationEngine.configurationFile=../default/itsi_correlation_engine.properties command.arg.5=-Dhttps.protocols=TLSv1.2,TLSv1.1 command.arg.6=-Dhttps.cipherSuites=TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256,TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384,TLS_RSA_WITH_AES_128_CBC_SHA256 chunked = true Update SA-ITSI-MetricAD/local/commands.conf with the following commands: [mad] type = custom command.arg.1=-J-Xmx1G command.arg.2=-Dlog4j.configurationFile=../default/log4j.xml command.arg.3=-Dlog4j2.threadContextMap=com.splunk.mad.util.MadThreadContextMapcommand.arg.4=-Dhttps.protocols=TLSv1.2,TLSv1.1 command.arg.5=-Dhttps.cipherSuites=TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256,TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384,TLS_RSA_WITH_AES_128_CBC_SHA256chunked = true |
2016-09-08 | ITSI-1268 | ITSI generates duplicate event_ids from the itsi_tracked_alerts index. This occurs when correlation search results contain an existing event_id. In this case, ITSI picks up the value of the event_id field and does not create a GUID for the event. Workaround: Rename the event_id field. |
2016-04-01 | ITSI-1346 | The 'Ping Host' action does not work when ITSI and Enterprise Security are installed on the same machine. Workaround: 1. Add the following stanza to $SPLUNK_HOME/etc/apps/SplunkEnterpriseSecurity/local/inputs.conf :
[app_imports_update://update_es] apps_to_update = (SA-(?!(ITOA|ITSI|IndexCreation|UserAccess)).*) | (Splunk_SA_.*)
2. Delete the "import = *" line from [] stanza of $SPLUNK_HOME/etc/apps/$APP/metadata/local.meta, where APP=SA-ITOA, SA-ITSI-ATAD, SA-ITSI-LicenseChecker, SA-IndexCreation, SA-UserAccess. |
Glass Table
Date filed | Issue number | Description |
---|---|---|
2019-07-08 | ITSI-3680 | You can't edit the title or description of existing beta glass tables from the glass table lister page. Workaround: Reload the GT lister page. |
2019-06-17 | ITSI-3505, SCP-13983 | Adding a drilldown from a Column or Area visualization causes infinite redirection to the drilldown link. |
2018-09-14 | ITSI-1567 | When you add a predictive model to a glass table, you cannot use the sparkline or trending value viz types because the prediction is a static value. |
KPI Search Calculation
Date filed | Issue number | Description |
---|---|---|
2018-04-26 | ITSI-248 | If an ITSI admin, who only has access to certain indexes, creates a KPI and uses the backfill option, the backfill runs through all data and not just the data that the admin has access to. |
Role Based Access Controls
Date filed | Issue number | Description |
---|---|---|
2019-03-29 | ITSI-2860 | If you assign the write_itsi_correlation_search capability to the itoa_analyst role, the role still cannot create a correlation search. Workaround: In addition to assigning the write_itsi_correlation_search capability to the itoa_analyst role, create a local.meta file at SPLUNK_HOME/etc/apps/itsi/metadata/ and add "itoa_analyst" to the [savedsearches] stanza.
For example: [savedsearches] access = read : [ * ], write: [ itoa_admin, itoa_team_admin, itoa_analyst ], delete: [ itoa_admin, itoa_team_admin, itoa_analyst ] export = system |
2018-04-26 | ITSI-248 | If an ITSI admin, who only has access to certain indexes, creates a KPI and uses the backfill option, the backfill runs through all data and not just the data that the admin has access to. |
Lister pages
Date filed | Issue number | Description |
---|---|---|
2019-07-08 | ITSI-3680 | You can't edit the title or description of existing beta glass tables from the glass table lister page. Workaround: Reload the GT lister page. |
2019-06-20 | ITSI-3588 | The Services lister page occasionally takes a long time to load. |
Service Analyzer
Date filed | Issue number | Description |
---|---|---|
2019-05-22 | ITSI-3258 | "HTTP 414: URI Too Long" when navigating in the ITSI UI. Workaround: ITSI does not limit URL length, so pages with too many characters fail to load. To work around this issue, limit your request lengths to the following:
|
2017-10-04 | ITSI-1290 | Filters with no matching results can't be saved in the Service Analyzer. |
Service Definition
Date filed | Issue number | Description |
---|---|---|
2016-03-28 | ITSI-1269 | On Windows 10 on Chrome, some selectors in the ITSI app do not function. |
Threshold Templates
Date filed | Issue number | Description |
---|---|---|
2019-04-08 | ITSI-2914 | When you first add a new KPI to a service template and apply Adaptive Thresholding, the additional KPI reuses the preview of the first KPI that was added to the template and displays misleading threshold values. Workaround: Once the scheduled daily adaptive threshold update runs, all KPIs linked to the template are correctly updated. Wait until midnight for the adaptive threshold values to update themselves. |
Predictive Analytics
Date filed | Issue number | Description |
---|---|---|
2019-03-20 | ITSI-2801 | Predictive Analytics occasionally fails to train models on Windows. Workaround: If search.log for the fit command reports the following error: ERROR ChunkedExternProcessor - stderr: ImportError: DLL load failed: The application has failed to start because its side-by-side configuration is incorrect. Please see the application event log or use the command-line sxstrace.exe tool for more detail. Then reinstall of Visual C++ 2008 runtime to resolve the issue: https://www.microsoft.com/en-in/download/details.aspx?id=40784 |
2018-09-14 | ITSI-1567 | When you add a predictive model to a glass table, you cannot use the sparkline or trending value viz types because the prediction is a static value. |
2018-08-01 | ITSI-1105 | After you delete a Predictive Analytics model through Lookups, the model still appears in the UI. |
Splunk App for Infrastructure Integration
Date filed | Issue number | Description |
---|---|---|
2019-05-21 | ITSI-3248 | The itoa_admin role does not have permission to create alerts in SAI. |
2018-09-24 | ITSI-1654 | Only 50,000 entities can be imported from the Splunk App for Infrastructure. Workaround: By default, the entity integration imports up to 50,000 entities from the Splunk App for Infrastructure. If you have more than 50,000 entities in Splunk App for Infrastructure, only the first 50,000 will be imported into ITSI. Increase the max_rows_per_query setting in $SPLUNK_HOME/etc/apps/SA-ITOA/local/limits.conf under the [kvstore] stanza to import more than 50,000 entities. |
Uncategorized issues
Date filed | Issue number | Description |
---|---|---|
2018-06-27 | ITSI-1287, ITSI-793 | Correlation searches created by manually editing savedsearches.conf do not appear on the correlation search lister page. Workaround: Do not create correlation searches by manually editing $SPLUNK_HOME/etc/apps/itsi/local/savedsearches.conf . The search will not appear on the correlation search lister page. Always create correlation searches directly in the IT Service Intelligence app. |
2015-12-01 | ITSI-1320 | When you install Enterprise Security on a search head with a pre-existing installation of ITSI, the ES-specific roles overwrite the ITSI-specific roles assigned to admin role. This disables access to all read/write objects in ITSI. Workaround: 1. In Splunk Web, go to Settings > Access Controls. 2. Select Roles > admin. |
2015-03-25 | ITSI-1293 | In a search head cluster environment, you cannot set up a recurring import (from CSV or search) through the UI. Workaround: 1. Create the modular input through the UI. ITSI adds the input as a new stanza in $SPLUNK_HOME/etc/apps/itsi/local/inputs.conf . It is not replicated across search peers.
Alternatively, if you're familiar with the format of modular inputs, you can create the input yourself. |
All ITSI Modules
Publication date | Issue number | Description |
---|---|---|
2017-03-21 | ITOA-7585 | When you bulk add services and an error caused by the racing condition occurs, the incorrect message "itsi_module does not exist" is displayed. |
2017-03-07 | MOD-979 | KPIs do not have consistent backfill settings across all modules. |
2017-01-17 | MOD-452 | The Analyze KPI button on the Service Details page is broken. |
2017-01-17 | MOD-402 | The Export to PDF option does not work in the drilldown to a module. |
2017-01-17 | MOD-296 | The extendable tab XML generator REST endpoint is located in DA-ITSI-OS instead of in common components where it can be used by all modules. |
2017-01-17 | MOD-591 | ITSI displays a misleading error message when a KPI template contains a field that cannot be resolved. |
2017-01-17 | MOD-498 | There is no upper limit to the number of characters a KPI title or description can contain. Long strings can negatively affect performance. |
2017-01-17 | MOD-309 | The Gruntfile.js included in ITSI modules uses double quotes instead of single quotes, which does not conform to the standard for all JavaScript files. |
2017-04-17 | MOD-2002 | When you drilldown from the Events tab, an "Invalid earliest_time" error occurs.
|
2017-01-17 | MOD-439 | Some modules do not have descriptions for saved searches. |
Application Server Module
Publication date | Issue number | Description |
---|---|---|
2017-01-27 | MOD-492 | If you reuse the same panel within a dashboard, the duplicate panel does not display any event data. |
Cloud Services Module
There are no known issues for this release.
Database Module
Publication date | Issue number | Description |
---|---|---|
2017-01-17 | MOD-586 | When a lookup is not configured for TA-Microsoft-SqlServer, ITSI displays a misleading error message on the server drilldown page. |
End User Experience Module
There are no known issues for this release.
Load Balancer Module
Publication date | Issue number | Description |
---|---|---|
2017-01-27 | MOD-492 | If you reuse the same panel within a dashboard, the duplicate panel does not display any event data. |
Operating System Module
Publication date | Issue number | Description |
---|---|---|
2017-04-13 | MOD-555 | The Storage Free Space % base search runs every minute while the Linux df command runs every 5 minutes. This causes data gaps. |
2017-04-10 | MOD-1964 | Windows data for memory free space is collected at different intervals than the Memory Free % KPI. |
2017-01-17 | MOD-1398 | Line, stack, and area charts do not display a metric gap when no metrics are available during a time period. |
Storage Module
There are no known issues for this release.
Virtualization Module
There are no known issues for this release.
Web Server Module
Publication date | Issue number | Description |
---|---|---|
2017-03-17 | MOD-320 | Some KPI ad hoc searches transform data with the stats command and do not retain time fields. The KPIs do not render anything and do not show thresholding details.
|
2017-03-17 | MOD-538 | When you add a new tab with panels and refresh the page, the page breaks. |
PREVIOUS Fixed issues in Splunk IT Service Intelligence |
NEXT Removed features in Splunk IT Service Intelligence |
This documentation applies to the following versions of Splunk® IT Service Intelligence: 4.3.0
Feedback submitted, thanks!