Splunk® IT Service Intelligence

Release Notes

Acrobat logo Download manual as PDF


Splunk IT Service Intelligence version 4.4.x will no longer be supported as of October 22, 2021. See the Splunk Software Support Policy for details. For information about upgrading to a supported version, see Before you upgrade IT Service Intelligence.
This documentation does not apply to the most recent version of ITSI. Click here for the latest version.
Acrobat logo Download topic as PDF

Known issues in Splunk IT Service Intelligence

IT Service Intelligence (ITSI) version 4.4.2 has the following known issues and workarounds.

Splunk platform issues that impact ITSI compatibility

Date filed Issue number Description
2019-02-14 SPL-155648
  • ITSI Event Analytics is incompatible with Splunk Enterprise version 7.2.0 - 7.2.3.
  • On versions 7.2.4 - 7.2.10, Event Analytics might duplicate events. To work around this issue, create a limits.conf file on all search heads at $SPLUNK_HOME/etc/apps/SA-ITOA/local/ and add the following stanza:
[search]
phased_execution_mode = auto
  • If you do not plan on using ITSI's Event Analytics functionality, the above does not apply.

See Splunk Enterprise system requirement in the Install and Upgrade Splunk IT Service Intelligence manual.

Adaptive Thresholding

Date filed Issue number Description
2019-12-07 ITSI-5035 Adaptive thresholds occasionally don't update at midnight.

Backup/Restore and Migration Issues

Date filed Issue number Description
2020-08-17 ITSI-10232 When restoring a backup, the action.itsi_event_generator.param.editor field is missing in the correlation search.

Workaround:
Upgrade to ITSI 4.7.0.

If using ITSI versions 4.4.5 and earlier, find the correlation search and remove it from the backup.

 

2020-05-13 ITSI-8269 The default scheduled backup runs twice in the same day.
2020-01-31 ITSI-5578 Upgrade from version 4.2.1 to 4.4.x or higher fails on importing comments because the new HEC token is missing 'itsi_group_comments_token'

Workaround:
1. Go to Settings > Data inputs on your ITSI instance. Locate HTTP Event Collector and verify that the "Token Value" field is empty.

2. If the token value is empty, delete the itsi_group_comments_token token through the UI and recreate it by disabling and reenabling the default_hec_initializer modular input at Settings > Data inputs > IT Service Intelligence HEC Initializer.

3. Disable and reenable the migration modular input at Settings > Data inputs > IT Service Intelligence Migration Modular Input.

2019-07-24 ITSI-3836 Objects such as service analyzers, glass tables, and deep dives are missing after upgrade.

Workaround:
If some objects are missing from the UI or unaccessible after you upgrade, the ACL objects corresponding to the objects might be missing or corrupted. For troubleshooting steps, see https://docs.splunk.com/Documentation/ITSI/latest/Install/Troubleshoot.
2019-07-23 ITSI-3830 Upon upgrade to 4.3.x, correlation searches and multi-KPI alerts are missing the 'all_info' field.
2019-05-07 ITSI-3119 Upgrade fails because a service template sync was queued.

Workaround:
Delete the backup using the curl command to change its status to Completed. Then force the service template sync. Restart Splunk software to complete the migration.
2019-01-03 ITSI-2164 ITSI backup times out due to an extremely large number of episode comments in the KV store.

Workaround:
Delete all comments prior to the backup (purge the collections in the KV store) or increase the Splunkd timeout and KV store limits. Then reduce the lifetime of the ITSI notable event collections in the KV store to archive them faster (the default is 6 months).
2018-10-16 ITSI-1748 You cannot restore an ITSI backup more than once.

Workaround:
This issue occurs because the saved search DA-ITSI-APM-EUEM_Base_Search is missing from the system. Create the missing saved search manually before restoring the backup. For example, create a local version of savedsearches.conf and add the following stanza:
[DA-ITSI-APM-EUEM_Base_Search]
 description =
 search =
 request.ui_dispatch_app = itsi
 request.ui_dispatch_view = search
 
2017-02-10 ITSI-1309 If multiple services use one KPI base search, and the total size of your services exceeds 50 MB, ITSI generates an error.

Workaround:
Increase the value for max_size_per_batch_save_mb (50MB is default) in $SPLUNK_HOME/etc/apps/SA-ITOA/local/limits.conf under the [kvstore] stanza. 
2016-05-02 ITSI-1305 After migration, shared objects (service analyzers, glass tables, and deep dives) are not accessible.

Workaround:
Use the curl command and create ACLs for each of the shared objects that are currently saved in the KV store collections: itsi_pages and itsi_service_analyzer.

For example:

$ curl -u admin:Splunk3r -k https://127.0.0.1:8089/servicesNS/nobody/SA-UserAccess/storage/collections/data/app_acl -X POST -H "Content-Type:application/json" -d '\{
"obj_id": "XXX-XXX-XXX",
"obj_type": "glass_table",
"obj_app": "itsi",
"obj_storename": "itsi_pages",

"obj_acl": \{
"obj_owner": "nobody",
"read": ["*"],
"write": ["*"],
"delete": ["*"]

},
"object_shared_by_inclusion": "true",
"acl_owner": "nobody"
}'
 

Bulk Import

Date filed Issue number Description
2020-08-26 ITSI-10391 The last line of a CSV file is not imported when importing services.

Workaround:
Open a file in Excel and save or save as and then import it. 
2020-02-05 ITSI-5623 KPI searches run into memory limits when trying to reconstruct entity filter causing fewer entities to be used than expected

Workaround:
Workaround is to increase memory limits for mvcombine and mvexpand splunk search commands. Default for both commands is set at 500, example is to double memory limit.
# Make an update to $SPLUNK_HOME/etc/system/local/limits.conf
# add the following stanzas:

[mvcombine]

max_mem_usage_mb = 1000

[mvexpand]

max_mem_usage_mb = 1000

 

2015-03-25 ITSI-1293 In a search head cluster environment, you cannot set up a recurring import (from CSV or search) through the UI.

Workaround:
1. Create the modular input through the UI. ITSI adds the input as a new stanza in $SPLUNK_HOME/etc/apps/itsi/local/inputs.conf. It is not replicated across search peers.

Alternatively, if you're familiar with the format of modular inputs, you can create the input yourself.
2.Copy the input stanza from the local version of inputs.conf and add it to shcluster/apps/itsi/local/inputs.conf on the deployer.
3. Let the deployer push the file to the search peers. The file is deployed to the default inputs.conf on each search peer.
4. Remove the modular input stanza from $SPLUNK_HOME/etc/apps/itsi/local/inputs.conf on the search head that created it. Otherwise it will take precedence on the deployer.

Deep Dive

Date filed Issue number Description
2020-03-17 ITSI-6288 The time picker in a deep dive disappears when clicked, making it impossible to set the time range.
2019-05-22 ITSI-3258 "HTTP 414: URI Too Long" when navigating in the ITSI UI.

Workaround:
ITSI does not limit URL length, so pages with too many characters fail to load. To work around this issue, limit your request lengths to the following:
  • Browser request: < 2048 characters
  • REST request: < 8192 characters.

Entities

Date filed Issue number Description
2015-02-12 ITSI-1286 When importing entities using Data inputs > IT Service Intelligence CSV Import, the page overflows.

Entity Rules

Date filed Issue number Description
2020-02-24 ITSI-5838 Imported entities aren't associated with the correct services or KPIs.

Notable Events

Date filed Issue number Description
2021-10-20 ITSI-19415 On Windows server, more than 1 rules engines processes are spawned at a time.

Workaround:
The root cause is the splunk phased_execution_mode. Edit the limits.conf file and add the line: 

[search] phased_execution_mode = auto
2020-10-22 ITSI-11620 Time-based action rules aren't triggered if the flow of events into an episode is paused.
2020-08-05 ITSI-9929 When the Rules Engine restarts, time-based policies with split by fields execute actions every minute.
2020-06-28 ITSI-9183 For time-based aggregation policies, new events are added to a broken episode after the Rules Engine restarts.
2020-06-08 ITSI-8855 ITSI rules engine nullpointer exception - and rules engine not grouping consistently - realtime issue
2020-02-27 ITSI-5932 ITSI doesn't support running Splunk Enterprise version 8.0.x with Ubuntu 18.04 and Open JDK 11.

Workaround:
Use Oracle JDK 11 or Open/Oracle JDK 8 instead of Open JDK 11, or use other versions of Linux.
2020-02-26 ITSI-5925 When the Rules Engine restarts it does a peer participation failure check before backfill process. If there are any failures, the Rules Engine restarts 3 times.
2020-01-29 ITSI-5498 "Add comment" episode actions are executed fewer times than expected.
2020-01-23 ITSI-5440 An error modal no longer appears if an episode action throws an error.
2019-12-18 ITSI-5113 The Aggregation Policy panel of the Event Analytics Monitoring dashboard doesn't display anything.
2019-11-25 ITSI-4954 When you upgrade to version 4.4.x, the migration script fails to add 'mod_time' to some objects in the 'itsi_notable_group_user' collection.

Workaround:
1. Add the field definition for mod_time in $SPLUNK_HOME/etc/apps/SA-ITOA/local/collections.conf:
 [itsi_notable_group_user]
 field.mod_time = time

[itsi_notable_group_system]
 field.mod_time = time
 

2. Update transforms.conf to include the fields event_identifier_hash, object_type, and mod_time.

Add the following stanza to $SPLUNK_HOME/etc/apps/SA-ITOA/local/transforms.conf:

 [itsi_notable_group_user_lookup]
 external_type = kvstore
 collection = itsi_notable_group_user
 fields_list = _key, status, severity, owner, event_identifier_hash, object_type, mod_time
 
 

3. Run the following search to update the mod_time for the itsi_notable_group_user collection:

|inputlookup itsi_notable_group_system_lookup
|fields _key, mod_time
|rename _key as id
|rename mod_time as sys_mod_time
|lookup itsi_notable_group_user_lookup _key as id OUTPUT owner severity status event_identifier_hash object_type mod_time
|rename id as _key
|eval mod_time=if(isnull(mod_time),sys_mod_time,mod_time)
|fields - sys_mod_time
|outputlookup itsi_notable_group_user_lookup

Once all the objects are in the itsi_notable_group_user collection with the mode_time, the next retention should delete the data.

4. After performing the workaround, revert the changes to collections.conf and transforms.conf.

2019-10-15 ITSI-4663 Upon upgrade to version 4.3.0 or later, the Rules Engine command fails with error: "Error occurred during initialization of VM".

Workaround:
This issue occurs because 32-bit Java cannot run the Rules Engine with the new memory settings introduced in version 4.3.x.
  1. Open or create a local copy of commands.conf at $SPLUNK_HOME/etc/apps/SA-ITOA/local.
  2. Add the following stanza:
    [itsirulesengine]
     command.arg.1=-J-Xmx1024M
     # reduced to 1024MB for 32 bit JDK/JRE
  3. Restart the Rules Engine, either by disabling and reenabling the itsi_event_grouping search, or by restarting Splunk software.

2019-10-09 ITSI-4606 ITSI backups keep failing and notable event KV store collections are growing very large.

Workaround:
This issue occurs because the indexed realtime search returns events over and over from buckets that use tsidx reduction. Disable tsidx reduction on the itsi_tracked_alerts and itsi_summary indexes and rebuild all old buckets on these indexes.
2019-01-03 ITSI-2164 ITSI backup times out due to an extremely large number of episode comments in the KV store.

Workaround:
Delete all comments prior to the backup (purge the collections in the KV store) or increase the Splunkd timeout and KV store limits. Then reduce the lifetime of the ITSI notable event collections in the KV store to archive them faster (the default is 6 months).
2018-12-10 ITSI-2059 Some notable events are added to more than one episode.

Workaround:
For an ITSI search head running Splunk 7.1 or 7.2, create or edit etc/system/local/limits.conf and add the following stanza: 
[search]
 phased_execution_mode = auto
 

For an ITSI search head running Splunk 7.3 or later, there is no need to change anything. 

2017-03-29 ITSI-1299 When your browser and the Splunk server are set to different DST time zones, the incorrect time might display for events in Episode Review.

Workaround:
Set your time zone to something other than "system default" even if you are in the same time zone as the system default.
2017-03-29 ITSI-1316 Splunkd connection fails due to "no_shared cipher matched" between client and server.

Workaround:
In order for notable event management and anomaly detection to work with Splunk platform 6.6, do the following:
  • Java 8/JRE 1.8/JDK 1.8*
* Download JCE 8 from http://www.oracle.com/technetwork/java/javase/downloads/jce8-download-2133166.html
* Unzip the downloaded file
* Place the two jars from the zip file into <java_jre_install_dir>/lib/security/ if running the JRE or <java_jdk_install_dir>/jre/lib/security if running the JDK.
  • Java 7/JRE 1.7/JDK 1.7*
* Download JCE 7 from http://www.oracle.com/technetwork/java/javase/downloads/jce-7-download-432124.html
* Unzip the downloaded file
* Place the two jars from the zip file into <java_jre_install_dir>/lib/security/ if running the JRE or <java_jdk_install_dir>/jre/lib/security if running the JDK.

Update SA-ITOA/local/commands.conf with the following commands: 

[itsirulesengine]

type = custom
command.arg.1=-J-Xmx1024M
command.arg.2=-Dlog4j.configurationFile=../default/log4j_rules_engine.xml
command.arg.3=-DitsiRulesEngine.configurationFile=../default/itsi_rules_engine.properties
command.arg.4=-Dhttps.protocols=TLSv1.2,TLSv1.1
command.arg.5=-Dhttps.cipherSuites=TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256,TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384,TLS_RSA_WITH_AES_128_CBC_SHA256
chunked = true

[itsicorrelationengine]

type = custom
command.arg.1=-J-Xmx1024M
command.arg.2=-Dlog4j.configurationFile=../default/log4j_correlation_engine.xml
command.arg.3=-J-XX:+UseConcMarkSweepGC
command.arg.4=-DitsiCorrelationEngine.configurationFile=../default/itsi_correlation_engine.properties
command.arg.5=-Dhttps.protocols=TLSv1.2,TLSv1.1
command.arg.6=-Dhttps.cipherSuites=TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256,TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384,TLS_RSA_WITH_AES_128_CBC_SHA256
chunked = true

Update SA-ITSI-MetricAD/local/commands.conf with the following commands:   

[mad]

type = custom
command.arg.1=-J-Xmx1G
command.arg.2=-Dlog4j.configurationFile=../default/log4j.xml
command.arg.3=-Dlog4j2.threadContextMap=com.splunk.mad.util.MadThreadContextMapcommand.arg.4=-Dhttps.protocols=TLSv1.2,TLSv1.1
command.arg.5=-Dhttps.cipherSuites=TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256,TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384,TLS_RSA_WITH_AES_128_CBC_SHA256chunked = true
2016-09-08 ITSI-1268 ITSI generates duplicate event_ids from the itsi_tracked_alerts index. This occurs when correlation search results contain an existing event_id. In this case, ITSI picks up the value of the event_id field and does not create a GUID for the event.

Workaround:
Rename the event_id field.
2016-04-01 ITSI-1346 The 'Ping Host' action does not work when ITSI and Enterprise Security are installed on the same machine.

Workaround:
1. Add the following stanza to $SPLUNK_HOME/etc/apps/SplunkEnterpriseSecurity/local/inputs.conf:
 [app_imports_update://update_es]
 apps_to_update = (SA-(?!(ITOA|ITSI|IndexCreation|UserAccess)).*) | (Splunk_SA_.*)


2. Delete the "import = *" line from [] stanza of $SPLUNK_HOME/etc/apps/$APP/metadata/local.meta, where APP=SA-ITOA, SA-ITSI-ATAD, SA-ITSI-LicenseChecker, SA-IndexCreation, SA-UserAccess.
3. Restart Splunk.

Notable Event Aggregation Policies

Date filed Issue number Description
2021-10-20 ITSI-19415 On Windows server, more than 1 rules engines processes are spawned at a time.

Workaround:
The root cause is the splunk phased_execution_mode. Edit the limits.conf file and add the line: 

[search] phased_execution_mode = auto
2020-10-22 ITSI-11620 Time-based action rules aren't triggered if the flow of events into an episode is paused.
2020-08-05 ITSI-9929 When the Rules Engine restarts, time-based policies with split by fields execute actions every minute.
2020-06-28 ITSI-9183 For time-based aggregation policies, new events are added to a broken episode after the Rules Engine restarts.
2020-06-08 ITSI-8855 ITSI rules engine nullpointer exception - and rules engine not grouping consistently - realtime issue
2020-02-27 ITSI-5932 ITSI doesn't support running Splunk Enterprise version 8.0.x with Ubuntu 18.04 and Open JDK 11.

Workaround:
Use Oracle JDK 11 or Open/Oracle JDK 8 instead of Open JDK 11, or use other versions of Linux.
2020-02-26 ITSI-5925 When the Rules Engine restarts it does a peer participation failure check before backfill process. If there are any failures, the Rules Engine restarts 3 times.
2020-01-29 ITSI-5498 "Add comment" episode actions are executed fewer times than expected.
2020-01-23 ITSI-5440 An error modal no longer appears if an episode action throws an error.
2019-12-18 ITSI-5113 The Aggregation Policy panel of the Event Analytics Monitoring dashboard doesn't display anything.
2019-11-25 ITSI-4954 When you upgrade to version 4.4.x, the migration script fails to add 'mod_time' to some objects in the 'itsi_notable_group_user' collection.

Workaround:
1. Add the field definition for mod_time in $SPLUNK_HOME/etc/apps/SA-ITOA/local/collections.conf:
 [itsi_notable_group_user]
 field.mod_time = time

[itsi_notable_group_system]
 field.mod_time = time
 

2. Update transforms.conf to include the fields event_identifier_hash, object_type, and mod_time.

Add the following stanza to $SPLUNK_HOME/etc/apps/SA-ITOA/local/transforms.conf:

 [itsi_notable_group_user_lookup]
 external_type = kvstore
 collection = itsi_notable_group_user
 fields_list = _key, status, severity, owner, event_identifier_hash, object_type, mod_time
 
 

3. Run the following search to update the mod_time for the itsi_notable_group_user collection:

|inputlookup itsi_notable_group_system_lookup
|fields _key, mod_time
|rename _key as id
|rename mod_time as sys_mod_time
|lookup itsi_notable_group_user_lookup _key as id OUTPUT owner severity status event_identifier_hash object_type mod_time
|rename id as _key
|eval mod_time=if(isnull(mod_time),sys_mod_time,mod_time)
|fields - sys_mod_time
|outputlookup itsi_notable_group_user_lookup

Once all the objects are in the itsi_notable_group_user collection with the mode_time, the next retention should delete the data.

4. After performing the workaround, revert the changes to collections.conf and transforms.conf.

2019-10-15 ITSI-4663 Upon upgrade to version 4.3.0 or later, the Rules Engine command fails with error: "Error occurred during initialization of VM".

Workaround:
This issue occurs because 32-bit Java cannot run the Rules Engine with the new memory settings introduced in version 4.3.x.
  1. Open or create a local copy of commands.conf at $SPLUNK_HOME/etc/apps/SA-ITOA/local.
  2. Add the following stanza:
    [itsirulesengine]
     command.arg.1=-J-Xmx1024M
     # reduced to 1024MB for 32 bit JDK/JRE
  3. Restart the Rules Engine, either by disabling and reenabling the itsi_event_grouping search, or by restarting Splunk software.

2019-10-09 ITSI-4606 ITSI backups keep failing and notable event KV store collections are growing very large.

Workaround:
This issue occurs because the indexed realtime search returns events over and over from buckets that use tsidx reduction. Disable tsidx reduction on the itsi_tracked_alerts and itsi_summary indexes and rebuild all old buckets on these indexes.
2019-01-03 ITSI-2164 ITSI backup times out due to an extremely large number of episode comments in the KV store.

Workaround:
Delete all comments prior to the backup (purge the collections in the KV store) or increase the Splunkd timeout and KV store limits. Then reduce the lifetime of the ITSI notable event collections in the KV store to archive them faster (the default is 6 months).
2018-12-10 ITSI-2059 Some notable events are added to more than one episode.

Workaround:
For an ITSI search head running Splunk 7.1 or 7.2, create or edit etc/system/local/limits.conf and add the following stanza: 
[search]
 phased_execution_mode = auto
 

For an ITSI search head running Splunk 7.3 or later, there is no need to change anything. 

2017-03-29 ITSI-1299 When your browser and the Splunk server are set to different DST time zones, the incorrect time might display for events in Episode Review.

Workaround:
Set your time zone to something other than "system default" even if you are in the same time zone as the system default.
2017-03-29 ITSI-1316 Splunkd connection fails due to "no_shared cipher matched" between client and server.

Workaround:
In order for notable event management and anomaly detection to work with Splunk platform 6.6, do the following:
  • Java 8/JRE 1.8/JDK 1.8*
* Download JCE 8 from http://www.oracle.com/technetwork/java/javase/downloads/jce8-download-2133166.html
* Unzip the downloaded file
* Place the two jars from the zip file into <java_jre_install_dir>/lib/security/ if running the JRE or <java_jdk_install_dir>/jre/lib/security if running the JDK.
  • Java 7/JRE 1.7/JDK 1.7*
* Download JCE 7 from http://www.oracle.com/technetwork/java/javase/downloads/jce-7-download-432124.html
* Unzip the downloaded file
* Place the two jars from the zip file into <java_jre_install_dir>/lib/security/ if running the JRE or <java_jdk_install_dir>/jre/lib/security if running the JDK.

Update SA-ITOA/local/commands.conf with the following commands: 

[itsirulesengine]

type = custom
command.arg.1=-J-Xmx1024M
command.arg.2=-Dlog4j.configurationFile=../default/log4j_rules_engine.xml
command.arg.3=-DitsiRulesEngine.configurationFile=../default/itsi_rules_engine.properties
command.arg.4=-Dhttps.protocols=TLSv1.2,TLSv1.1
command.arg.5=-Dhttps.cipherSuites=TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256,TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384,TLS_RSA_WITH_AES_128_CBC_SHA256
chunked = true

[itsicorrelationengine]

type = custom
command.arg.1=-J-Xmx1024M
command.arg.2=-Dlog4j.configurationFile=../default/log4j_correlation_engine.xml
command.arg.3=-J-XX:+UseConcMarkSweepGC
command.arg.4=-DitsiCorrelationEngine.configurationFile=../default/itsi_correlation_engine.properties
command.arg.5=-Dhttps.protocols=TLSv1.2,TLSv1.1
command.arg.6=-Dhttps.cipherSuites=TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256,TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384,TLS_RSA_WITH_AES_128_CBC_SHA256
chunked = true

Update SA-ITSI-MetricAD/local/commands.conf with the following commands:   

[mad]

type = custom
command.arg.1=-J-Xmx1G
command.arg.2=-Dlog4j.configurationFile=../default/log4j.xml
command.arg.3=-Dlog4j2.threadContextMap=com.splunk.mad.util.MadThreadContextMapcommand.arg.4=-Dhttps.protocols=TLSv1.2,TLSv1.1
command.arg.5=-Dhttps.cipherSuites=TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256,TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384,TLS_RSA_WITH_AES_128_CBC_SHA256chunked = true
2016-09-08 ITSI-1268 ITSI generates duplicate event_ids from the itsi_tracked_alerts index. This occurs when correlation search results contain an existing event_id. In this case, ITSI picks up the value of the event_id field and does not create a GUID for the event.

Workaround:
Rename the event_id field.
2016-04-01 ITSI-1346 The 'Ping Host' action does not work when ITSI and Enterprise Security are installed on the same machine.

Workaround:
1. Add the following stanza to $SPLUNK_HOME/etc/apps/SplunkEnterpriseSecurity/local/inputs.conf:
 [app_imports_update://update_es]
 apps_to_update = (SA-(?!(ITOA|ITSI|IndexCreation|UserAccess)).*) | (Splunk_SA_.*)


2. Delete the "import = *" line from [] stanza of $SPLUNK_HOME/etc/apps/$APP/metadata/local.meta, where APP=SA-ITOA, SA-ITSI-ATAD, SA-ITSI-LicenseChecker, SA-IndexCreation, SA-UserAccess.
3. Restart Splunk.

Glass Table

Date filed Issue number Description
2020-02-12 ITSI-5749 When you upgrade a glass table to the beta framework, KPI icons lose their color.
2019-10-10 ITSI-4630, SCP-18188 Beta glass tables ignore the latest time setting in the time range picker.
2019-10-09 ITSI-4617 KPI and ad hoc searches in beta glass tables are run under the context "App: Search" instead of "App: ITSI". This prevents certain app-restricted searches such as Predictive Analytics searches from running.
2019-09-23 ITSI-4439, SCP-18415 The title and description requirement for visualizations, shapes, and icons makes object selection difficult.
2019-09-11 ITSI-4353 Single-value radial visualizations on beta glass tables are not grayed out when an associated service is in maintenance mode.
2018-09-14 ITSI-1567 When you add a predictive model to a glass table, you cannot use the sparkline or trending value viz types because the prediction is a static value.

KPI Base Searches

Date filed Issue number Description
2020-01-07 ITSI-5220 Shared base searches are generating thousands of "Broken Socket" messages but ITSI functionality is not impacted.

KPI Search Calculation

Date filed Issue number Description
2020-10-20 ITSI-11551 KPI alerting stops working after upgrade.
2020-09-25 ITSI-11102 KPI alerting fails to create notable events when the KPI severity changes to and from N/A.

Workaround:
Comment out the following line in $SPLUNK_HOME/etc/apps/SA-ITOA/bin/command_set_severity_fields.py
to disable the backfill check:
self._check_historical_data(result.get('_time'))
As a side effect, KPI alerting might trigger automatically during initial backfill.
2020-06-12 ITSI-8920 The Service Analyzer side panel shows a blank severity instead of "Unknown" when an entity's value is N/A.

Maintenance Window

Date filed Issue number Description
2020-10-07 ITSI-11354 The maintenance window UI calculation of the daylight saving starting day is incorrect.

Workaround:
Check the start time displayed as a preview (in UTC) when creating maintenance windows to ensure that your maintenance window is created correctly. 

Role Based Access Controls

Date filed Issue number Description
2019-03-29 ITSI-2860 If you assign the write_itsi_correlation_search capability to the itoa_analyst role, the role still cannot create a correlation search.

Workaround:
In addition to assigning the write_itsi_correlation_search capability to the itoa_analyst role, create a local.meta file at SPLUNK_HOME/etc/apps/itsi/metadata/ and add "itoa_analyst" to the [savedsearches] stanza.

For example:

[savedsearches]
access = read : [ * ], write: [ itoa_admin, itoa_team_admin, itoa_analyst ], delete: [ itoa_admin, itoa_team_admin, itoa_analyst ]
export = system

Service Analyzer

Date filed Issue number Description
2020-06-23 ITSI-9085 Saved Service Analyzer views with wildcard filters don't load correctly.
2020-06-12 ITSI-8920 The Service Analyzer side panel shows a blank severity instead of "Unknown" when an entity's value is N/A.
2020-05-27 ITSI-8551 Using special characters in KPI names breaks the service health monitoring search.

Workaround:
Use regular apostrophe character.
2020-03-03 ITSI-6004 The severities shown in the Service Analyzer tree don't match what's displayed in the side panel.
2020-02-03 ITSI-5596 Real-time searches don't return results in the Service Analyzer, but the real-time search window options are still available in the time picker.

Workaround:
You can automatically refresh the Service Analyzer when a relative time range is selected from the time picker as opposed to real-time. See https://docs.splunk.com/Documentation/ITSI/latest/User/ServiceAnalyzer#Automatically_refresh_the_Service_Analyzer.

 

2019-09-19 ITSI-4424 The itoa_admin role cannot permanently dismiss the cyclic dependency warning.
2019-05-22 ITSI-3258 "HTTP 414: URI Too Long" when navigating in the ITSI UI.

Workaround:
ITSI does not limit URL length, so pages with too many characters fail to load. To work around this issue, limit your request lengths to the following:
  • Browser request: < 2048 characters
  • REST request: < 8192 characters.

2017-10-04 ITSI-1290 Filters with no matching results can't be saved in the Service Analyzer.

Service Definition

Date filed Issue number Description
2016-03-28 ITSI-1269 On Windows 10 on Chrome, some selectors in the ITSI app do not function.

Predictive Analytics

Date filed Issue number Description
2019-10-09 ITSI-4617 KPI and ad hoc searches in beta glass tables are run under the context "App: Search" instead of "App: ITSI". This prevents certain app-restricted searches such as Predictive Analytics searches from running.
2019-03-20 ITSI-2801 Predictive Analytics occasionally fails to train models on Windows.

Workaround:
If search.log for the fit command reports the following error:

ERROR ChunkedExternProcessor - stderr: ImportError: DLL load failed: The application has failed to start because its side-by-side configuration is incorrect. Please see the application event log or use the command-line sxstrace.exe tool for more detail.

To resolve this issue, reinstall of Visual C++ 2008 runtime: [1]

2018-09-14 ITSI-1567 When you add a predictive model to a glass table, you cannot use the sparkline or trending value viz types because the prediction is a static value.

Splunk App for Infrastructure Integration

Date filed Issue number Description
2019-05-21 ITSI-3248 The itoa_admin role does not have permission to create alerts in SAI.
2018-09-24 ITSI-1654 Only 50,000 entities can be imported from the Splunk App for Infrastructure.

Workaround:
By default, the entity integration imports up to 50,000 entities from the Splunk App for Infrastructure. If you have more than 50,000 entities in Splunk App for Infrastructure, only the first 50,000 will be imported into ITSI. Increase the max_rows_per_query setting in $SPLUNK_HOME/etc/apps/SA-ITOA/local/limits.conf under the [kvstore] stanza to import more than 50,000 entities.

Uncategorized issues

Date filed Issue number Description
2021-02-23 ITSI-14414 ITSI internal licenses remove customer's Dev-Test licenses since dev-test licenses cannot stack with other license types

Workaround:
* Disable itsi_license_checker_unbundle input
* Re-install Dev/Test licenses
* Restart splunk
* Keep license_checker_unbundle modular input disabled while Dev/Test license is being used
** Warn customer that without internal license, they may see increased indexing usage on their Dev/Test license
2020-12-16 ITSI-12692 ITSI upgrade failed due to "Sort operation used more than the maximum 33554432 bytes of RAM"

Workaround:
The ITSI migration from older version to 4.4.* or later fails, with error in itsi_upgrade.log:
'An error occurred. (Internal read failed with error code \'96\' and message \'Executor error during find command :: caused by :: errmsg: "Sort operation used more than the maximum 33554432 bytes of RAM. Add an index, or specify a smaller limit."\')'}]
Root cause is likely large collections too costly sort in memory, usually the itsi_services collection. The workaround is to add acceleration to the collection SA-ITOA/local/collections.conf: 

[itsi_services]

accelerated_fields._key = \{"_key": 1}
2020-12-14 ITSI-12561 ServiceNow add-on is unable to create incidents or alerts if there are unescaped double quotes in the search command.

Workaround:
There are 2 workarounds:
# If feasible, don't use the double quotes in the fields' values.
# In *notable_event_actions.conf*, make the *alt_command_supported_version* under the *[snow_incident]* stanza later that the current ServiceNow version.

If the alt_command_supported_version is higher than the ServiceNow's current version, the older search command will be used, which escapes the double quotes.

2020-10-29 ITSI-11724 The entire entity import fails when an existing entity references a non-existent service, declaring unhelpful error.
2020-09-29 ITSI-11199, ITSI-9901 In Ubuntu OS, ITSI modular inputs create duplicate instances when restarted using the modular input UI.

Workaround:
Set start_by_shell to "true" in inputs.conf for all ITSI modular inputs. For more information, see https://docs.splunk.com/Documentation/Splunk/latest/AdvancedDev/ModInputsScripts#Override_default_run_behavior_for_modular_input_scripts
2020-09-28 ITSI-11173 "Unable to retrieve Java version on this machine, make sure Java is properly installed" on Windows 2016.
2020-08-31 ITSI-10492 The KPI performance runtime headroom calculation on the ITSI Health Check dashboard is incorrect for search head clusters.

Workaround:
Upgrade to ITSI version 4.7.0. Alternatively, on older versions, clone the dashboard and then edit the copy to make the first "| stats count AS run_count ..." with "| stats dc(sid) AS run_count ..." only report on the unique number of invocations for each search.
2020-07-02 ITSI-9275 If a saved search generated for a KPI metric search has "<number>-<number>", the eval thinks '-' is an operator rather than part of key string and the search throws an error.

Workaround:
Manually enclose the eval value in quotes.
2020-05-06 ITSI-8090 When ITSI generates a closing event but can't pull the title and description values from the last event, the KV store values and the UI are updated with %title% and %description% instead.
2020-04-22 ITSI-7008 The "Send to Phantom" episode action fails with exception "could not convert string to float"

Workaround:
Set the exec_delay_time setting to a value larger than 28 seconds in $SPLUNK_HOME/etc/apps/SA-ITOA/local/inputs.conf for all itsi_notable_event_actions_queue_consumer inputs.

However, this change introduces an additional delay to notable event processing, which delays sending notable event information to Phantom.

2020-04-16 ITSI-6933 You can't dispatch a Phantom alert action using Python 3 on Splunk 8.0.0 or higher.

Workaround:
Make a local copy of alert_actions.conf at $SPLUNK_HOME/etc/apps/SA-ITOA/local/alert_actions.conf and add the following setting:
[itsi_event_action_send_to_phantom]
python.version = python2
 
2020-02-27 ITSI-5933 When you try to create a ServiceNow incident from an episode that has a ticket already assigned to it, the modal doesn't appear or takes a long time to load.
2020-02-04 ITSI-5615 Token based authentication failing to authenticate on Search-Head-Cluster for SA-ITOA REST API interfaces

Workaround:
When calling rest endpoint, for example:
curl --insecure -H "Authorization: Bearer -X GET  <TOKEN>" https://localhost:8089/servicesNS/nobody/SA-ITOA/itoa_interface/v4.3.1/service/

You will see messages like this:

{"message":"(500, '[HTTP 401] Client is not authenticated')"}.

This issue is caused by SPL-183142.

Only affects SH clusters but not standalone instance, from 7.3 onwards. Specifically impacts apps that use rest endpoints. The workaround is to use the old way of authenticating, username/password.

2020-01-29 ITSI-5553 When you restore any service from a partial backup, the restore changes the thresholds of other services.
2020-01-13 ITSI-5257 In the Operating System Module, some panels on the "OS Host Details" dashboard don't populate for Windows data.

Workaround:
Add the setting eval pid=coalesce(pid,ID_Process) to the following files in $SPLUNK_HOME/etc/apps/DA-ITSI-OS/default/data/ui/panels:


* cpu_top_consumers_list.xml
* cpu_usage_overall.xml
* memory_top_consumers_list.xml
* overview_cpu.xml
2019-12-11 ITSI-5055 When you import services through a service template, the KPIs are recreated, causing issues with anomaly detection and backfill.
2019-11-11 ITSI-4813 Upon upgrade to 4.4.x, splunkd.log reports the following error message: "ImportError: No module named itsi_path".

Workaround:
1. Install SA-ITOA on all license masters in your environment.

2. On each license master, disable all inputs in $SPLUNK_HOME/etc/apps/SA-ITOA/local/inputs.conf by setting disabled=1 under each individual stanza. The following example file disables all inputs:

[itsi_user_access_init]
disabled = 1

[itsi_user_access_init://upgrade_capabilities]
disabled = 1

[configure_itsi://splunko]
disabled = 1

[itsi_upgrade://itsi_migration]
disabled = 1

[itsi_refresher]
disabled = 1

[itsi_refresher://consistency]
disabled = 1

[itsi_consumer://consumer1]
disabled = 1

[itsi_backup_restore://itsi_backup_restore]
disabled = 1

[itsi_scheduled_backup_caller://itsi_scheduled_backup]
disabled = 1

[itsi_service_template_update_scheduler://itsi_service_template_update_scheduler]
disabled = 1

[itsi_backfill]
disabled = 1

[itsi_backfill://backfiller]
disabled = 1

[itsi_async_csv_loader]
disabled = 1

[itsi_async_csv_loader://async_csv_loader]
disabled = 1

[itsi_notable_event_archive://age_notable_event]
disabled = 1

[maintenance_minder://populate_operative_maintenance_log]
disabled = 1

[itsi_default_aggregation_policy_loader://default_policy_loader]
disabled = 1

[itsi_default_correlation_search_acl_loader://default_acl_loader]
disabled = 1

[itsi_notable_event_hec_init://default_hec_initializer]
disabled = 1

[itsi_notable_event_actions_queue_consumer://alpha]
disabled = 1

[itsi_notable_event_actions_queue_consumer://beta]
disabled = 1

[itsi_notable_event_actions_queue_consumer://gamma]
disabled = 1

[itsi_notable_event_actions_queue_consumer://zeta]
disabled = 1

[itsi_notable_event_actions_queue_consumer://delta]
disabled = 1

[http]
disabled = 1

[script://$SPLUNK_HOME/etc/apps/SA-ITOA/bin/import_icons_SA_ITOA.py]
disabled = 1

[itsi_entity_exchange_consumer://itsi_entity_exchange_consumer1]
disabled = 1

[itsi_age_kpi_alert_value_cache://age_kpi_alert_value_cache]
disabled = 1

[monitor://$SPLUNK_HOME/var/run/splunk/dispatch/*/itsi_search*]
disabled = 1
 
2019-07-31 ITSI-3902 There are excessive InsecureRequestWarning messages in splunkd.log when using Python 2 libraries.

Workaround:
Migrate ITSI and Splunk Enterprise to Python 3. For instructions, see https://docs.splunk.com/Documentation/ITSI/latest/Install/Python3.
2019-07-01 ITSI-3666 Upon upgrade, the Splunk product name changes from Splunk>enterprise to Splunk>hunk.

Workaround:
Ensure you have active group defined in server.conf

[license]

active_group = Enterprise

2019-06-08 ITSI-3437 Correlation searches don't work with real-time searches.
2019-05-30 ITSI-3322 If you add a correlation search in ITSI which contains a sub-search returning into an eval, you get a message "Invalid search string: This search cannot be parsed when parse_only is set to true."

Workaround:
You can't use a sub-search returning into an eval in a correlation search. As a workaround, create and save a basic correlation search with all of the information you want outside of the search. Then as an admin user, go to Settings > Searches, reports, and alerts and open the correlation search you just created. Add the sub-search you were trying to add there.
2019-02-12 ITSI-2471 If ITSI is installed on multiple environments with multiple license masters, and any indexer interacts with both environments, a duplicate licensing error occurs because both environments have the same auto-generated ITSI license stack.

Workaround:
Follow the workaround described in the deployment planning docs for the version of ITSI you're currently using: https://docs.splunk.com/Documentation/ITSI/latest/Install/Plan#ITSI_license_requirements
2018-06-27 ITSI-1287, ITSI-793 Correlation searches created by manually editing savedsearches.conf do not appear on the correlation search lister page.

Workaround:
Do not create correlation searches by manually editing $SPLUNK_HOME/etc/apps/itsi/local/savedsearches.conf. The search will not appear on the correlation search lister page. Always create correlation searches directly in the IT Service Intelligence app.
2015-12-01 ITSI-1320 When you install Enterprise Security on a search head with a pre-existing installation of ITSI, the ES-specific roles overwrite the ITSI-specific roles assigned to admin role. This disables access to all read/write objects in ITSI.

Workaround:
1. In Splunk Web, go to Settings > Access Controls.

2. Select Roles > admin.
3. Add itoa_admin, itoa_analyst, and itoa_user to Selected roles.
4. Click Save.

All ITSI Modules

Publication date Issue number Description
2017-03-21 ITOA-7585 When you bulk add services and an error caused by the racing condition occurs, the incorrect message "itsi_module does not exist" is displayed.
2017-03-07 MOD-979 KPIs do not have consistent backfill settings across all modules.
2017-01-17 MOD-452 The Analyze KPI button on the Service Details page is broken.
2017-01-17 MOD-402 The Export to PDF option does not work in the drilldown to a module.
2017-01-17 MOD-296 The extendable tab XML generator REST endpoint is located in DA-ITSI-OS instead of in common components where it can be used by all modules.
2017-01-17 MOD-591 ITSI displays a misleading error message when a KPI template contains a field that cannot be resolved.
2017-01-17 MOD-498 There is no upper limit to the number of characters a KPI title or description can contain. Long strings can negatively affect performance.
2017-01-17 MOD-309 The Gruntfile.js included in ITSI modules uses double quotes instead of single quotes, which does not conform to the standard for all JavaScript files.
2017-04-17 MOD-2002 When you drilldown from the Events tab, an "Invalid earliest_time" error occurs.


Workaround:
Disable drilldown from the Events tab.

2017-01-17 MOD-439 Some modules do not have descriptions for saved searches.

Application Server Module

Publication date Issue number Description
2017-01-27 MOD-492 If you reuse the same panel within a dashboard, the duplicate panel does not display any event data.

Cloud Services Module

There are no known issues for this release.

Database Module

Publication date Issue number Description
2017-01-17 MOD-586 When a lookup is not configured for TA-Microsoft-SqlServer, ITSI displays a misleading error message on the server drilldown page.

End User Experience Module

There are no known issues for this release.

Load Balancer Module

Publication date Issue number Description
2017-01-27 MOD-492 If you reuse the same panel within a dashboard, the duplicate panel does not display any event data.

Operating System Module

Publication date Issue number Description
2017-04-13 MOD-555 The Storage Free Space % base search runs every minute while the Linux df command runs every 5 minutes. This causes data gaps.
2017-04-10 MOD-1964 Windows data for memory free space is collected at different intervals than the Memory Free % KPI.
2017-01-17 MOD-1398 Line, stack, and area charts do not display a metric gap when no metrics are available during a time period.

Storage Module

There are no known issues for this release.

Virtualization Module

There are no known issues for this release.

Web Server Module

Publication date Issue number Description
2017-03-17 MOD-320 Some KPI ad hoc searches transform data with the stats command and do not retain time fields. The KPIs do not render anything and do not show thresholding details.
2017-03-17 MOD-538 When you add a new tab with panels and refresh the page, the page breaks.
Last modified on 03 November, 2021
PREVIOUS
Fixed issues in Splunk IT Service Intelligence
  NEXT
Removed features in Splunk IT Service Intelligence

This documentation applies to the following versions of Splunk® IT Service Intelligence: 4.4.2


Was this documentation topic helpful?

You must be logged into splunk.com in order to post comments. Log in now.

Please try to keep this discussion focused on the content covered in this documentation topic. If you have a more general question about Splunk functionality or are experiencing a difficulty with Splunk, consider posting a question to Splunkbase Answers.

0 out of 1000 Characters