Splunk® IT Service Intelligence

Release Notes

Acrobat logo Download manual as PDF


Splunk IT Service Intelligence version 4.0.x reached its End of Life on January 19, 2021. See the Splunk Software Support Policy for details. For information about upgrading to a supported version, see Plan an upgrade of IT Service Intelligence.
This documentation does not apply to the most recent version of Splunk® IT Service Intelligence. Click here for the latest version.
Acrobat logo Download topic as PDF

Known issues in Splunk IT Service Intelligence

IT Service Intelligence (ITSI) version 4.0.4 has the following known issues and workarounds.

Splunk platform issues that impact ITSI compatibility

Date filed Issue number Description
2019-02-14 SPL-155648
  • ITSI Event Analytics is incompatible with Splunk Enterprise version 7.2.0 - 7.2.3.
  • On versions 7.0.5 - 7.1.x and 7.2.4 - 7.2.10, event analytics might duplicate events. To work around this issue, create a limits.conf file on all search heads at $SPLUNK_HOME/etc/apps/SA-ITOA/local/ and add the following stanza:
[search]
phased_execution_mode = auto
  • If you do not plan on using event analytics, the above does not apply.

Backup/Restore and Migration Issues

Date filed Issue number Description
2019-07-24 ITSI-3836 Objects such as service analyzers, glass tables, and deep dives are missing after upgrade.

Workaround:
If some objects are missing from the UI or unaccessible after you upgrade, the ACL objects corresponding to the objects might be missing or corrupted. For troubleshooting steps, see https://docs.splunk.com/Documentation/ITSI/latest/Install/Troubleshoot.
2019-05-07 ITSI-3119 Upgrade fails because a service template sync was queued.

Workaround:
Delete the backup using the curl command to change its status to Completed. Then force the service template sync. Restart Splunk software to complete the migration.
2019-03-11 ITSI-2714 In a search head cluster environment, the Backup/Restore page only lets you download local nightly backups. It does not display a list of all other backup files on all instances.
2019-01-03 ITSI-2164 ITSI backup times out due to an extremely large number of episode comments in the KV store.

Workaround:
Delete all comments prior to the backup (purge the collections in the KV store) or increase the Splunkd timeout and KV store limits. Then reduce the lifetime of the ITSI notable event collections in the KV store to archive them faster (the default is 6 months).
2018-10-04 ITSI-1681 Restoring a file fails if Splunk is installed on a path that contains "." (for example, /opt/splunk-7.2.0)
2017-08-14 ITSI-1349 You can't restore a shared glass table from a partial backup.

Workaround:
Before creating the partial backup, set the permissions on the shared glass table to private. After restoring, change the permissions back to shared.
2017-02-10 ITSI-1309 If multiple services use one KPI base search, and the total size of your services exceeds 50 MB, ITSI generates an error.

Workaround:
Increase the value for max_size_per_batch_save_mb (50MB is default) in $SPLUNK_HOME/etc/apps/SA-ITOA/local/limits.conf under the [kvstore] stanza. 
2016-05-02 ITSI-1305 After migration, shared objects (service analyzers, glass tables, and deep dives) are not accessible.

Workaround:
Use the curl command and create ACLs for each of the shared objects that are currently saved in the KV store collections: itsi_pages and itsi_service_analyzer.

For example: 

$ curl -u admin:Splunk3r -k https://127.0.0.1:8089/servicesNS/nobody/SA-UserAccess/storage/collections/data/app_acl -X POST -H "Content-Type:application/json" -d '\{
"obj_id": "XXX-XXX-XXX",
"obj_type": "glass_table",
"obj_app": "itsi",
"obj_storename": "itsi_pages",

"obj_acl": \{
"obj_owner": "nobody",
"read": ["*"],
"write": ["*"],
"delete": ["*"]

},
"object_shared_by_inclusion": "true",
"acl_owner": "nobody"
}'
 

Bulk Import

Date filed Issue number Description
2015-03-25 ITSI-1293 In a search head cluster environment, you cannot set up a recurring import (from CSV or search) through the UI.

Workaround:
1. Create the modular input through the UI. ITSI adds the input as a new stanza in $SPLUNK_HOME/etc/apps/itsi/local/inputs.conf. It is not replicated across search peers.

Alternatively, if you're familiar with the format of modular inputs, you can create the input yourself.
2.Copy the input stanza from the local version of inputs.conf and add it to shcluster/apps/itsi/local/inputs.conf on the deployer.
3. Let the deployer push the file to the search peers. The file is deployed to the default inputs.conf on each search peer.
4. Remove the modular input stanza from $SPLUNK_HOME/etc/apps/itsi/local/inputs.conf on the search head that created it. Otherwise it will take precedence on the deployer.

Deep Dive

Date filed Issue number Description
2018-09-13 ITSI-1556 When you drill down to a deep dive from the Predictive Analytics dashboard in Internet Explorer, the deep dive opens with no lanes because the URL is too long.

Workaround:
Manually add the KPI lanes to the deep dive.
2016-12-14 ITSI-525 If you zoom in on a specific time range in a deep dive while using twin-lane comparison, the comparisons that appear are occasionally offset by up to a minute.

Entities

Date filed Issue number Description
2019-02-19 ITSI-2540 The curl command to delete all entities times out with a large amount of entities.
2015-02-12 ITSI-1286 When importing entities using Data inputs > IT Service Intelligence CSV Import, the page overflows.

Entity Rules

Date filed Issue number Description
2019-04-17 ITSI-2967 The "does not match" entity rule acts as if it has a wildcard at the end of the string, filtering out all possibilities that start with the value rather than just that value.

Workaround:
Create an OR condition in the entity rules of services such that the logic works.

Rule 1: location does not match A, B, C, ..., Z

OR

Rule 2: location matches ZZZ

Notable Events

Date filed Issue number Description
2020-02-27 ITSI-5932 ITSI doesn't support running Splunk Enterprise version 8.0.x with Ubuntu 18.04 and Open JDK 11.

Workaround:
Use Oracle JDK 11 or Open/Oracle JDK 8 instead of Open JDK 11, or use other versions of Linux.
2019-11-20 ITSI-4940 Nothing blocks you from creating an external ticket from an episode for which a ticket was already created.
2019-02-04 ITSI-2396 If multiple episodes are created by the same aggregation policy and you try to close both of them, only one episode is closed and the other remains open.
2019-01-18 ITSI-2310 The backfilling system does not handle boundary conditions in a reasonable way. This causes events to be ignored or duplicated.
2019-01-03 ITSI-2164 ITSI backup times out due to an extremely large number of episode comments in the KV store.

Workaround:
Delete all comments prior to the backup (purge the collections in the KV store) or increase the Splunkd timeout and KV store limits. Then reduce the lifetime of the ITSI notable event collections in the KV store to archive them faster (the default is 6 months).
2019-01-02 ITSI-2161 The "Add column" option in the View Settings of Episodes Review does not display custom fields.
2018-12-18 ITSI-2128 Episodes that have been closed can be marked as open when the Rules Engine restarts, so new events are added to them.

Workaround:
Add the following property to $SPLUNK_HOME/etc/apps/SA-ITOA/local/itsi_rules_engine.properties: 
active_group_info_search = search `itsi_event_management_group_index` \
  | stats max(itsi_group_count) as itsi_group_count \
    values(itsi_is_last_event) as itsi_is_last_event \
    max(itsi_last_event_time) as itsi_last_event_time \
    values(itsi_parent_group_id) as itsi_parent_group_id \
    values(itsi_policy_id) as itsi_policy_id \
    values(itsi_split_by_hash) as itsi_split_by_hash \
    values(itsi_first_event_id) as itsi_first_event_id \
    values(itsi_first_event_time) as itsi_first_event_time \
    values(itsi_group_assignee) as itsi_group_assignee \
    values(itsi_group_description) as itsi_group_description \
    values(itsi_group_severity) as itsi_group_severity \
    values(itsi_group_status) as itsi_group_status \
    values(itsi_group_title) as itsi_group_title by itsi_group_id \
  | join itsi_group_id [|inputlookup itsi_notable_group_system_lookup \
    | eval itsi_group_id=_key | fields itsi_group_id is_active] | where is_active=1

The macro identifies open groups by checking if is_active=1 for backfill.

2018-11-20 ITSI-1956, ITSI-2362 If you perform an action on an episode in an environment where multiple ITSI search heads are sharing the same indexers, the action does not appear on the Activity tab of the episode.
2018-11-07 ITSI-1910 Notable events fields have no length safeguard and can exceed the default Splunk limits for parsing, causing the events to be unusable.

Workaround:
Raise the truncate limits to at least 100,000.

Example props.conf settings: 

[itsi_notable:event]
 KV_MODE = none
 INDEXED_EXTRACTIONS = JSON
 TRUNCATE=100000
[itsi_notable:group]
 KV_MODE = none
 INDEXED_EXTRACTIONS = JSON
 TRUNCATE=100000

2018-08-23 ITSI-1381 Aggregation policy action rules occasionally pass empty results to episode actions.

Workaround:
Event data occasionally fails to pass to an episode action if the notable event aggregation policy action rule is configured to trigger off of the first event in the episode. This might intermittently cause the action to run while the first event in an episode is still being indexed. To increase the delay between when the action is triggered and when it runs:

1. Create a copy of inputs.conf in $SPLUNK_HOME/etc/apps/SA-ITOA/local.

2. For each of the [itsi_notable_event_actions_queue_consumer://<*>] stanzas, increase the value of the 'exec_delay_time' setting. For example: 

[itsi_notable_event_actions_queue_consumer://alpha]
 exec_delay_time = 1

[itsi_notable_event_actions_queue_consumer://beta]
 exec_delay_time = 1

[itsi_notable_event_actions_queue_consumer://gamma]
 exec_delay_time = 1

[itsi_notable_event_actions_queue_consumer://zeta]
 exec_delay_time = 1

[itsi_notable_event_actions_queue_consumer://delta]
 exec_delay_time = 1

The exec_delay_time needed might vary depending on your Splunk installation and configuration.

2017-03-29 ITSI-1299 When your browser and the Splunk server are set to different DST time zones, the incorrect time might display for events in Episode Review.

Workaround:
Set your time zone to something other than "system default" even if you are in the same time zone as the system default.
2017-03-29 ITSI-1316 Splunkd connection fails due to "no_shared cipher matched" between client and server.

Workaround:
In order for notable event management and anomaly detection to work with Splunk platform 6.6, do the following:
  • Java 8/JRE 1.8/JDK 1.8*
* Download JCE 8 from http://www.oracle.com/technetwork/java/javase/downloads/jce8-download-2133166.html
* Unzip the downloaded file
* Place the two jars from the zip file into <java_jre_install_dir>/lib/security/ if running the JRE or <java_jdk_install_dir>/jre/lib/security if running the JDK.
  • Java 7/JRE 1.7/JDK 1.7*
* Download JCE 7 from http://www.oracle.com/technetwork/java/javase/downloads/jce-7-download-432124.html
* Unzip the downloaded file
* Place the two jars from the zip file into <java_jre_install_dir>/lib/security/ if running the JRE or <java_jdk_install_dir>/jre/lib/security if running the JDK.

Update SA-ITOA/local/commands.conf with the following commands: 

[itsirulesengine] 

type = custom
command.arg.1=-J-Xmx1024M
command.arg.2=-Dlog4j.configurationFile=../default/log4j_rules_engine.xml
command.arg.3=-DitsiRulesEngine.configurationFile=../default/itsi_rules_engine.properties
command.arg.4=-Dhttps.protocols=TLSv1.2,TLSv1.1
command.arg.5=-Dhttps.cipherSuites=TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256,TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384,TLS_RSA_WITH_AES_128_CBC_SHA256
chunked = true

[itsicorrelationengine] 

type = custom
command.arg.1=-J-Xmx1024M
command.arg.2=-Dlog4j.configurationFile=../default/log4j_correlation_engine.xml
command.arg.3=-J-XX:+UseConcMarkSweepGC
command.arg.4=-DitsiCorrelationEngine.configurationFile=../default/itsi_correlation_engine.properties
command.arg.5=-Dhttps.protocols=TLSv1.2,TLSv1.1
command.arg.6=-Dhttps.cipherSuites=TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256,TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384,TLS_RSA_WITH_AES_128_CBC_SHA256
chunked = true

Update SA-ITSI-MetricAD/local/commands.conf with the following commands:   

[mad] 

type = custom
command.arg.1=-J-Xmx1G
command.arg.2=-Dlog4j.configurationFile=../default/log4j.xml
command.arg.3=-Dlog4j2.threadContextMap=com.splunk.mad.util.MadThreadContextMapcommand.arg.4=-Dhttps.protocols=TLSv1.2,TLSv1.1
command.arg.5=-Dhttps.cipherSuites=TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256,TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384,TLS_RSA_WITH_AES_128_CBC_SHA256chunked = true
2016-09-08 ITSI-1268 ITSI generates duplicate event_ids from the itsi_tracked_alerts index. This occurs when correlation search results contain an existing event_id. In this case, ITSI picks up the value of the event_id field and does not create a GUID for the event.

Workaround:
Rename the event_id field.
2016-04-01 ITSI-1346 The 'Ping Host' action does not work when ITSI and Enterprise Security are installed on the same machine.

Workaround:
1. Add the following stanza to $SPLUNK_HOME/etc/apps/SplunkEnterpriseSecurity/local/inputs.conf: 
 [app_imports_update://update_es]
 apps_to_update = (SA-(?!(ITOA|ITSI|IndexCreation|UserAccess)).*) | (Splunk_SA_.*)



2. Delete the "import = *" line from [] stanza of $SPLUNK_HOME/etc/apps/$APP/metadata/local.meta, where APP=SA-ITOA, SA-ITSI-ATAD, SA-ITSI-LicenseChecker, SA-IndexCreation, SA-UserAccess.

3. Restart Splunk.

Notable Event Aggregation Policies

Date filed Issue number Description
2020-02-27 ITSI-5932 ITSI doesn't support running Splunk Enterprise version 8.0.x with Ubuntu 18.04 and Open JDK 11.

Workaround:
Use Oracle JDK 11 or Open/Oracle JDK 8 instead of Open JDK 11, or use other versions of Linux.
2019-11-20 ITSI-4940 Nothing blocks you from creating an external ticket from an episode for which a ticket was already created.
2019-02-04 ITSI-2396 If multiple episodes are created by the same aggregation policy and you try to close both of them, only one episode is closed and the other remains open.
2019-01-18 ITSI-2310 The backfilling system does not handle boundary conditions in a reasonable way. This causes events to be ignored or duplicated.
2019-01-03 ITSI-2164 ITSI backup times out due to an extremely large number of episode comments in the KV store.

Workaround:
Delete all comments prior to the backup (purge the collections in the KV store) or increase the Splunkd timeout and KV store limits. Then reduce the lifetime of the ITSI notable event collections in the KV store to archive them faster (the default is 6 months).
2019-01-02 ITSI-2161 The "Add column" option in the View Settings of Episodes Review does not display custom fields.
2018-12-18 ITSI-2128 Episodes that have been closed can be marked as open when the Rules Engine restarts, so new events are added to them.

Workaround:
Add the following property to $SPLUNK_HOME/etc/apps/SA-ITOA/local/itsi_rules_engine.properties: 
active_group_info_search = search `itsi_event_management_group_index` \
  | stats max(itsi_group_count) as itsi_group_count \
    values(itsi_is_last_event) as itsi_is_last_event \
    max(itsi_last_event_time) as itsi_last_event_time \
    values(itsi_parent_group_id) as itsi_parent_group_id \
    values(itsi_policy_id) as itsi_policy_id \
    values(itsi_split_by_hash) as itsi_split_by_hash \
    values(itsi_first_event_id) as itsi_first_event_id \
    values(itsi_first_event_time) as itsi_first_event_time \
    values(itsi_group_assignee) as itsi_group_assignee \
    values(itsi_group_description) as itsi_group_description \
    values(itsi_group_severity) as itsi_group_severity \
    values(itsi_group_status) as itsi_group_status \
    values(itsi_group_title) as itsi_group_title by itsi_group_id \
  | join itsi_group_id [|inputlookup itsi_notable_group_system_lookup \
    | eval itsi_group_id=_key | fields itsi_group_id is_active] | where is_active=1

The macro identifies open groups by checking if is_active=1 for backfill.

2018-11-20 ITSI-1956, ITSI-2362 If you perform an action on an episode in an environment where multiple ITSI search heads are sharing the same indexers, the action does not appear on the Activity tab of the episode.
2018-11-07 ITSI-1910 Notable events fields have no length safeguard and can exceed the default Splunk limits for parsing, causing the events to be unusable.

Workaround:
Raise the truncate limits to at least 100,000.

Example props.conf settings: 

[itsi_notable:event]
 KV_MODE = none
 INDEXED_EXTRACTIONS = JSON
 TRUNCATE=100000
[itsi_notable:group]
 KV_MODE = none
 INDEXED_EXTRACTIONS = JSON
 TRUNCATE=100000

2018-08-23 ITSI-1381 Aggregation policy action rules occasionally pass empty results to episode actions.

Workaround:
Event data occasionally fails to pass to an episode action if the notable event aggregation policy action rule is configured to trigger off of the first event in the episode. This might intermittently cause the action to run while the first event in an episode is still being indexed. To increase the delay between when the action is triggered and when it runs:

1. Create a copy of inputs.conf in $SPLUNK_HOME/etc/apps/SA-ITOA/local.

2. For each of the [itsi_notable_event_actions_queue_consumer://<*>] stanzas, increase the value of the 'exec_delay_time' setting. For example: 

[itsi_notable_event_actions_queue_consumer://alpha]
 exec_delay_time = 1

[itsi_notable_event_actions_queue_consumer://beta]
 exec_delay_time = 1

[itsi_notable_event_actions_queue_consumer://gamma]
 exec_delay_time = 1

[itsi_notable_event_actions_queue_consumer://zeta]
 exec_delay_time = 1

[itsi_notable_event_actions_queue_consumer://delta]
 exec_delay_time = 1

The exec_delay_time needed might vary depending on your Splunk installation and configuration.

2017-03-29 ITSI-1299 When your browser and the Splunk server are set to different DST time zones, the incorrect time might display for events in Episode Review.

Workaround:
Set your time zone to something other than "system default" even if you are in the same time zone as the system default.
2017-03-29 ITSI-1316 Splunkd connection fails due to "no_shared cipher matched" between client and server.

Workaround:
In order for notable event management and anomaly detection to work with Splunk platform 6.6, do the following:
  • Java 8/JRE 1.8/JDK 1.8*
* Download JCE 8 from http://www.oracle.com/technetwork/java/javase/downloads/jce8-download-2133166.html
* Unzip the downloaded file
* Place the two jars from the zip file into <java_jre_install_dir>/lib/security/ if running the JRE or <java_jdk_install_dir>/jre/lib/security if running the JDK.
  • Java 7/JRE 1.7/JDK 1.7*
* Download JCE 7 from http://www.oracle.com/technetwork/java/javase/downloads/jce-7-download-432124.html
* Unzip the downloaded file
* Place the two jars from the zip file into <java_jre_install_dir>/lib/security/ if running the JRE or <java_jdk_install_dir>/jre/lib/security if running the JDK.

Update SA-ITOA/local/commands.conf with the following commands: 

[itsirulesengine] 

type = custom
command.arg.1=-J-Xmx1024M
command.arg.2=-Dlog4j.configurationFile=../default/log4j_rules_engine.xml
command.arg.3=-DitsiRulesEngine.configurationFile=../default/itsi_rules_engine.properties
command.arg.4=-Dhttps.protocols=TLSv1.2,TLSv1.1
command.arg.5=-Dhttps.cipherSuites=TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256,TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384,TLS_RSA_WITH_AES_128_CBC_SHA256
chunked = true

[itsicorrelationengine] 

type = custom
command.arg.1=-J-Xmx1024M
command.arg.2=-Dlog4j.configurationFile=../default/log4j_correlation_engine.xml
command.arg.3=-J-XX:+UseConcMarkSweepGC
command.arg.4=-DitsiCorrelationEngine.configurationFile=../default/itsi_correlation_engine.properties
command.arg.5=-Dhttps.protocols=TLSv1.2,TLSv1.1
command.arg.6=-Dhttps.cipherSuites=TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256,TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384,TLS_RSA_WITH_AES_128_CBC_SHA256
chunked = true

Update SA-ITSI-MetricAD/local/commands.conf with the following commands:   

[mad] 

type = custom
command.arg.1=-J-Xmx1G
command.arg.2=-Dlog4j.configurationFile=../default/log4j.xml
command.arg.3=-Dlog4j2.threadContextMap=com.splunk.mad.util.MadThreadContextMapcommand.arg.4=-Dhttps.protocols=TLSv1.2,TLSv1.1
command.arg.5=-Dhttps.cipherSuites=TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256,TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384,TLS_RSA_WITH_AES_128_CBC_SHA256chunked = true
2016-09-08 ITSI-1268 ITSI generates duplicate event_ids from the itsi_tracked_alerts index. This occurs when correlation search results contain an existing event_id. In this case, ITSI picks up the value of the event_id field and does not create a GUID for the event.

Workaround:
Rename the event_id field.
2016-04-01 ITSI-1346 The 'Ping Host' action does not work when ITSI and Enterprise Security are installed on the same machine.

Workaround:
1. Add the following stanza to $SPLUNK_HOME/etc/apps/SplunkEnterpriseSecurity/local/inputs.conf: 
 [app_imports_update://update_es]
 apps_to_update = (SA-(?!(ITOA|ITSI|IndexCreation|UserAccess)).*) | (Splunk_SA_.*)



2. Delete the "import = *" line from [] stanza of $SPLUNK_HOME/etc/apps/$APP/metadata/local.meta, where APP=SA-ITOA, SA-ITSI-ATAD, SA-ITSI-LicenseChecker, SA-IndexCreation, SA-UserAccess.

3. Restart Splunk.

Glass Table

Date filed Issue number Description
2018-09-14 ITSI-1567 When you add a predictive model to a glass table, you cannot use the sparkline or trending value viz types because the prediction is a static value.
2017-08-14 ITSI-1349 You can't restore a shared glass table from a partial backup.

Workaround:
Before creating the partial backup, set the permissions on the shared glass table to private. After restoring, change the permissions back to shared.

KPI Base Searches

Date filed Issue number Description
2017-04-13 ITSI-1294 KPI base search does not handle duplicate entity aliases, causing incorrect group KPI statistics.

Workaround:
1. When Splunk detects duplicate aliases, a warning message appears in the Messages menu. Click Show duplicates to open the ITSI Health Check dashboard which lists the entities with duplicate aliases. (Or click Dashboards > ITSI Health Check from the ITSI main menu.)

2. Click Configure > Entities and edit the entity definitions with duplicate aliases. Keep the alias value for one of the entities and edit the other to remove the duplicate alias value.

Note: You can also merge the duplicates by moving all the fields that differ to one entity, then deleting the extra copy.

3. Turn off all module entity discovery searches.

 

Maintenance Window

Date filed Issue number Description
2017-08-08 ITSI-1236 When you navigate back and forth in the Edit Maintenance Window modal, some information is populated incorrectly.

Performance

Date filed Issue number Description
2019-07-11 ITSI-3731 For pre-4.2.0 versions on Splunk Enterprise version 7.1.x, the service template lister page takes a long time to load when templates are linked to a lot of services.

Workaround:
Either upgrade Splunk Enterprise to version 7.2 or later, or upgrade ITSI to version 4.2.0 or later.


Role Based Access Controls

Date filed Issue number Description
2019-03-29 ITSI-2860 If you assign the write_itsi_correlation_search capability to the itoa_analyst role, the role still cannot create a correlation search.

Workaround:
In addition to assigning the write_itsi_correlation_search capability to the itoa_analyst role, create a local.meta file at SPLUNK_HOME/etc/apps/itsi/metadata/ and add "itoa_analyst" to the [savedsearches] stanza.

For example: 

[savedsearches]
access = read : [ * ], write: [ itoa_admin, itoa_team_admin, itoa_analyst ], delete: [ itoa_admin, itoa_team_admin, itoa_analyst ]
export = system

2018-02-06 ITSI-440 When itoa_admin, itoa_analyst, itoa_team_admin and itoa_user roles are added to a new custom role, users assigned to the custom role do not have the "edit permissions" capability for saved service analyzers.
2017-10-16 ITSI-437 Roles inheriting from itoa_admin do not behave like itoa_admin. For example, the inheriting role cannot edit permissions on pages such as glass tables, deep dives, and service analyzers.

Workaround:
Make the user a member of the itoa_admin role (rather than just a member of a role inheriting from it).

Service Analyzer

Date filed Issue number Description
2019-02-21 ITSI-2562 Backend Service Analyzer searches ignore filtering and calculate statistics against all services, leading to significantly longer than expected search times and higher memory usage
2017-10-04 ITSI-1290 Filters with no matching results can't be saved in the Service Analyzer.

Service Definition

Date filed Issue number Description
2016-03-28 ITSI-1269 On Windows 10 on Chrome, some selectors in the ITSI app do not function.
2015-11-13 ITSI-1266 ITSI does not work as expected on Windows due to memory issues on the host machine.

Service Templates

Date filed Issue number Description
2019-07-11 ITSI-3731 For pre-4.2.0 versions on Splunk Enterprise version 7.1.x, the service template lister page takes a long time to load when templates are linked to a lot of services.

Workaround:
Either upgrade Splunk Enterprise to version 7.2 or later, or upgrade ITSI to version 4.2.0 or later.


Teams

Date filed Issue number Description
2019-03-25 ITSI-2822 When you filter services on the team details page, no services match the filter.

Workaround:
Type the filter using only lower case characters. 

Predictive Analytics

Date filed Issue number Description
2019-10-01 ITSI-4530, ITSI-4604 The KPI Predictions chart on the Predictive Analytics dashboard does not display the correct timestamps.
2019-10-01 ITSI-4531 The Predictive Analytics Dashboard "KPI Predictions" panel plots results in GMT rather than the user's timezone.
2019-01-18 ITSI-2309 Predictive Analytics is not available if ITSI is installed on Splunk Enterprise version 7.0.x.

Workaround:
Perform one of the following workarounds: 
A. Upgrade to Splunk version 7.1.x or later.

B. If you cannot upgrade, modify the Predictive Analytics macros: 

1. Navigate to $SPLUNK_HOME/etc/apps/SA-ITOA/local
2. Create or edit a macros.conf file.
3. Add the following stanza to the file:

# Macro to train KPI trend models and health score KPI relations.
[train_kpi_trends(2)]
args=sid,suffix
definition = `itsi_predictive_analytics_dataset($sid$)`\
  | appendpipe [fit LinearRegression fit_intercept=true now_avg_hs from\
    "value_avg:*" into app:itsi_predict_kpi_hs_$suffix$ | fields - _time *]\
  | fit StandardScaler "value_*" with_mean=true with_std=true into app:itsi_predict_kpi_ss_$suffix$\
  | `prepare_kpi_trend_data($sid$,$suffix$)`\
  | map search="| inputcsv itsi_predict_kpi_$suffix$.csv | fit GradientBoostingRegressor \"next30mkpi_$kpiid$\" from\
    \"SS_*\" \"this_date_*\" \"last30mkpi_$kpiid$\" \"value_avg: $kpiid$\" into app:itsi_predict_kpi_$model_suffix$"\
    maxsearches=100\
  | head 1\
  | fields "predicted(*)"\
  | rename "predicted(next30mkpi_*)" as *\
  | fields - _time\
  | foreach * [eval <<FIELD>>=1]\
  | untable modelname kpi dummyfield\
  | fields - dummyfield\
  | eval modelname="itsi_predict_kpi_".replace(kpi, "-", "_")\
  | append [| listmodels\
    | search name="itsi_predict_kpi_*_$suffix$"\
    | rename name as modelname\
    | fields modelname]

4. Save the file and restart Splunk. 

5. Verify the fix by training a predictive model for a small time period (like 7 days).
2018-12-03 ITSI-2000 The Service Health Score and KPIs Over Time graph fails to report duplicate KPI names when multiple dependent services have identical KPIs.
2018-10-04 ITSI-1680 The "Analyze in Deep Dive" option on the Predictive Analytics dashboard does not work for some services.
2018-09-14 ITSI-1567 When you add a predictive model to a glass table, you cannot use the sparkline or trending value viz types because the prediction is a static value.
2018-09-13 ITSI-1556 When you drill down to a deep dive from the Predictive Analytics dashboard in Internet Explorer, the deep dive opens with no lanes because the URL is too long.

Workaround:
Manually add the KPI lanes to the deep dive.
2018-08-14 ITSI-1160 "Error in 'fit' command: Invalid model name" when you try to save a Predictive Analytics model. This error occurs because ITSI and MLTK have different rules for naming conventions.
2018-08-01 ITSI-1105 After you delete a Predictive Analytics model through Lookups, the model still appears in the UI.
2018-07-24 ITSI-1027, ITSI-1098 The Predictive Analytics fit command sometimes fails with the following error: "Unexpected end of JSON input".

Splunk App for Infrastructure Integration

Date filed Issue number Description
2018-09-24 ITSI-1654 Only 50,000 entities can be imported from the Splunk App for Infrastructure.

Workaround:
By default, the entity integration imports up to 50,000 entities from the Splunk App for Infrastructure. If you have more than 50,000 entities in Splunk App for Infrastructure, only the first 50,000 will be imported into ITSI. Increase the max_rows_per_query setting in $SPLUNK_HOME/etc/apps/SA-ITOA/local/limits.conf under the [kvstore] stanza to import more than 50,000 entities.

Uncategorized issues

Date filed Issue number Description
2019-08-23 ITSI-4171 When your system's time zone and the Splunk time zone set in your user preferences are different, it may cause several hours of lag between Rules Engine logs and Python logs in the _internal index.

Workaround:
Configure your Splunk time zone to be the same as your system's time zone.
2019-08-05 ITSI-3924 An error in the multi-KPI "status over time" alert search results in the percentages always been 100%.

Workaround:
This error occurs because the stats count is creating the field occurrences, but the getPercentage macro is expecting the field occurrence. To work around this issue, open the multi-KPI alert in the correlation search editor and change the word "occurrence" to "occurrences". Note that this action prevents you from using the Multi-KPI Alerts page to edit the correlation search in the future.
2019-07-01 ITSI-3666 Upon upgrade, the Splunk product name changes from Splunk>enterprise to Splunk>hunk.

Workaround:
Ensure you have active group defined in server.conf

[license]

active_group = Enterprise

2019-02-12 ITSI-2471 If ITSI is installed on multiple environments with multiple license masters, and any indexer interacts with both environments, a duplicate licensing error occurs because both environments have the same auto-generated ITSI license stack.

Workaround:
Follow the workaround described in the deployment planning docs for the version of ITSI you're currently using: https://docs.splunk.com/Documentation/ITSI/latest/Install/Plan#ITSI_license_requirements
2018-12-05 ITSI-2022, ITSI-2010 The ITSI SDK version 1.0 is not fully compatible with ITSI 4.x due to refactoring of grouped events (episodes).

Workaround:
Because individual notable events are immutable in version 4.0.0 and later, you must pass the unique identifier of an episode (itsi_group_id) instead of the identifier of an individual notable event. Therefore, the previously event_id parameter is now itsi_group_id.
2018-11-16 ITSI-1941 When you create a multi-KPI alert, the summary index stores the entity_title as the search head and not the entity used to populate the data.

Workaround:
Create a correlation search as an alternative to a multi-KPI alert.

1. Click Configure > Correlation Searches.

2. Click Create New Search > Create Correlation Search.

3. Provide a search name.

4. Enter a search that contains the service ID. For example, `mka_sn_kpin("Password Reset Tool","CPU Utilization: %")`.

5. Enter a notable event title and description. For example, %service_name% degraded because of %entity_title%.

6. Configure other fields and click Save to save the correlation search.

7. Go to Episode Review and you should start seeing events.

2018-07-25 ITSI-1332, SPL-157799 In the KPI configuration, if you click in the generated search to run the search in a separate tab, no results are displayed and no errors are logged because the search process has crashed.

Workaround:
This issue is fixed in Splunk Enterprise version 7.1.3. Upgrade to Splunk Enterprise 7.1.3 to avoid the issue.
2018-06-27 ITSI-1287, ITSI-793 Correlation searches created by manually editing savedsearches.conf do not appear on the correlation search lister page.

Workaround:
Do not create correlation searches by manually editing $SPLUNK_HOME/etc/apps/itsi/local/savedsearches.conf. The search will not appear on the correlation search lister page. Always create correlation searches directly in the IT Service Intelligence app.
2016-09-09 ITSI-1336 All requests to KV Store that exceed 50MB are truncated. This can cause issues with backup, migration, access control, and service loading on UI pages.

Workaround:
In SA-ITOA/local/limits.conf, in the [kvstore] stanza, increase the value of max_size_per_result_mb. The default 100MB value supports 2,000 KPIs. If you have more than 2,000 KPIs, increase the value accordingly.
2015-12-01 ITSI-1320 When you install Enterprise Security on a search head with a pre-existing installation of ITSI, the ES-specific roles overwrite the ITSI-specific roles assigned to admin role. This disables access to all read/write objects in ITSI.

Workaround:
1. In Splunk Web, go to Settings > Access Controls.

2. Select Roles > admin.
3. Add itoa_admin, itoa_analyst, and itoa_user to Selected roles.
4. Click Save.

All ITSI Modules

Publication date Issue number Description
2017-03-21 ITOA-7585 When you bulk add services and an error caused by the racing condition occurs, the incorrect message "itsi_module does not exist" is displayed.
2017-03-07 MOD-979 KPIs do not have consistent backfill settings across all modules.
2017-01-17 MOD-452 The Analyze KPI button on the Service Details page is broken.
2017-01-17 MOD-402 The Export to PDF option does not work in the drilldown to a module.
2017-01-17 MOD-296 The extendable tab XML generator REST endpoint is located in DA-ITSI-OS instead of in common components where it can be used by all modules.
2017-01-17 MOD-591 ITSI displays a misleading error message when a KPI template contains a field that cannot be resolved.
2017-01-17 MOD-498 There is no upper limit to the number of characters a KPI title or description can contain. Long strings can negatively affect performance.
2017-01-17 MOD-309 The Gruntfile.js included in ITSI modules uses double quotes instead of single quotes, which does not conform to the standard for all JavaScript files.
2017-04-17 MOD-2002 When you drilldown from the Events tab, an "Invalid earliest_time" error occurs.


Workaround:
Disable drilldown from the Events tab.

2017-01-17 MOD-439 Some modules do not have descriptions for saved searches.

Application Server Module

Publication date Issue number Description
2017-01-27 MOD-492 If you reuse the same panel within a dashboard, the duplicate panel does not display any event data.

Cloud Services Module

There are no known issues for this release.

Database Module

Publication date Issue number Description
2017-01-17 MOD-586 When a lookup is not configured for TA-Microsoft-SqlServer, ITSI displays a misleading error message on the server drilldown page.

End User Experience Module

There are no known issues for this release.

Load Balancer Module

Publication date Issue number Description
2017-01-27 MOD-492 If you reuse the same panel within a dashboard, the duplicate panel does not display any event data.

Operating System Module

Publication date Issue number Description
2017-04-13 MOD-555 The Storage Free Space % base search runs every minute while the Linux df command runs every 5 minutes. This causes data gaps.
2017-04-10 MOD-1964 Windows data for memory free space is collected at different intervals than the Memory Free % KPI.
2017-01-17 MOD-1398 Line, stack, and area charts do not display a metric gap when no metrics are available during a time period.

Storage Module

There are no known issues for this release.

Virtualization Module

There are no known issues for this release.

Web Server Module

Publication date Issue number Description
2017-03-17 MOD-320 Some KPI ad hoc searches transform data with the stats command and do not retain time fields. The KPIs do not render anything and do not show thresholding details.
2017-03-17 MOD-538 When you add a new tab with panels and refresh the page, the page breaks.
Last modified on 13 August, 2022
PREVIOUS
Fixed issues in Splunk IT Service Intelligence 		  NEXT
Credits

This documentation applies to the following versions of Splunk® IT Service Intelligence: 4.0.4

Was this documentation topic helpful?


You must be logged into splunk.com in order to post comments. Log in now.

Please try to keep this discussion focused on the content covered in this documentation topic. If you have a more general question about Splunk functionality or are experiencing a difficulty with Splunk, consider posting a question to Splunkbase Answers.

0 out of 1000 Characters