Splunk® IT Service Intelligence

Release Notes

Download manual as PDF

This documentation does not apply to the most recent version of ITSI. Click here for the latest version.
Download topic as PDF

Known issues in Splunk IT Service Intelligence

IT Service Intelligence (ITSI) version 4.2.1 has the following known issues and workarounds.

Splunk platform issues that impact ITSI compatibility

Date filed Issue number Description
2019-02-14 SPL-155648
  • ITSI Event Analytics is incompatible with Splunk Enterprise version 7.2.0 - 7.2.3.
  • On versions 7.1.x and 7.2.4 - 7.2.9, Event Analytics might duplicate events. To work around these issues, create a limits.conf file on all search heads at $SPLUNK_HOME/etc/apps/SA-ITOA/local/ and add the following stanza:
[search]
phased_execution_mode = auto
  • If you do not plan on using Event Analytics, the above does not apply.

See Splunk Enterprise system requirement in the Install and Upgrade Splunk IT Service Intelligence manual.

Backup/Restore and Migration Issues

Date filed Issue number Description
2019-11-20 ITSI-4917 During backup/restore, notable event archiving fails for large KV store collections, causing the restore to be very slow.

Workaround:
Check whether ITSI shows the following error message in the internal logs at source=itsi_notable_event_archive-age_notable_event.log:
 2019-12-06 10:39:06,951 ERROR [itsi.notable_event_archive] [__init__] [exception] [15399] [HTTP 500] Splunkd internal error; [{'type': 'ERROR', 'code': None, 'text': 'An error occurred. (Internal read failed with error code \'96\' and message \'Executor error during OP_QUERY find :: caused by :: errmsg: "Sort operation used more than the maximum 33554432 bytes of RAM. Add an index, or specify a smaller limit."\')'}] 

If so, add acceleration on the large collections to reduce the memory pressure during a sort. Open $SPLUNK_HOME/etc/apps/SA-ITOA/local/collections.conf and add the following stanzas for the impacted collections:

#if the collection group_system is too large
[itsi_notable_group_system]
accelerated_fields.mod_time = {"mod_time": 1}

# if the group_user is too large
[itsi_notable_group_user] 
accelerated_fields.mod_time = {"mod_time": 1}

2019-06-11 ITSI-3448, ITSI-277 The backup/restore UI does not take daylight savings time into account.
2019-06-11 ITSI-3452 Upon upgrade to 4.2.x in a search head cluster, the event grouping custom command "itsirulesengine" may fail to run on some search heads: "ERROR Unable to invoke factory method in class class org.apache.logging.log4j.core.config.PropertiesPlugin".

Workaround:
To validate the root cause, log in to each search head and run the following search: | itsirulesengine
If the search fails on a search head, an error message appears in the UI and in the search.log. 
Once you have identified the offending search head, perform one of the following actions:
1. SSH to the search head and remove the following files:
cd /opt/splunk/etc/apps/SA-ITOA/lib/java/event_management/libs
 rm akka-actor_2.11-2.3.15.jar akka-slf4j_2.11-2.3.15.jar config-1.2.1.jar log4j-api-2.3.jar log4j-core-2.3.jar log4j-slf4j-impl-2.3.jar scala-library-2.11.5.jar slf4j-api-1.7.21.jar
Then retry the search.
2. If the files were pushed from the deployer, go to the deployer and remove the files:
cd /opt/splunk/shcluster/apps/SA-ITOA/lib/java/event_management/libs
 rm akka-actor_2.11-2.3.15.jar akka-slf4j_2.11-2.3.15.jar config-1.2.1.jar log4j-api-2.3.jar log4j-core-2.3.jar log4j-slf4j-impl-2.3.jar scala-library-2.11.5.jar slf4j-api-1.7.21.jar 
Then push the bundle to the search head and retry the search.
2019-05-24 ITSI-3292 Upgrade to 4.2.x on a search head cluster fails with a 414 error because of large pre-existing savedsearches.conf files.

Workaround:
1. Stop the search heads.

2. Delete the $SPLUNK_HOME/etc/apps/itsi/local/savedsearches.conf on each search head. If you have non-ITSI searches (custom searches, correlations searches, etc.) you can also remove just the "indicators" searches to trim it.
3. Restart the search head cluster.
4. Apply the backup from the UI.

2019-05-07 ITSI-3119 Upgrade fails because a service template sync was queued.

Workaround:
Delete the backup using the curl command to change its status to Completed. Then force the service template sync. Restart Splunk software to complete the migration.
2019-05-02 ITSI-3081 Partial backup does not list services associated with beta glass tables.

Workaround:
Navigate to your beta glass table(s), identify the services that your glass table visualizations are associated with, and manually add those services to the backup.
2019-03-11 ITSI-2714 In a search head cluster environment, the Backup/Restore page only lets you download local nightly backups. It does not display a list of all other backup files on all instances.
2019-01-03 ITSI-2164 ITSI backup times out due to an extremely large number of episode comments in the KV store.

Workaround:
Delete all comments prior to the backup (purge the collections in the KV store) or increase the Splunkd timeout and KV store limits. Then reduce the lifetime of the ITSI notable event collections in the KV store to archive them faster (the default is 6 months).
2018-10-16 ITSI-1748 You cannot restore an ITSI backup more than once.

Workaround:
This issue occurs because the saved search DA-ITSI-APM-EUEM_Base_Search is missing from the system. Create the missing saved search manually before restoring the backup. For example, create a local version of savedsearches.conf and add the following stanza:
[DA-ITSI-APM-EUEM_Base_Search]
 description =
 search =
 request.ui_dispatch_app = itsi
 request.ui_dispatch_view = search
 
2017-02-10 ITSI-1309 If multiple services use one KPI base search, and the total size of your services exceeds 50 MB, ITSI generates an error.

Workaround:
Increase the value for max_size_per_batch_save_mb (50MB is default) in $SPLUNK_HOME/etc/apps/SA-ITOA/local/limits.conf under the [kvstore] stanza. 
2016-05-02 ITSI-1305 After migration, shared objects (service analyzers, glass tables, and deep dives) are not accessible.

Workaround:
Use the curl command and create ACLs for each of the shared objects that are currently saved in the KV store collections: itsi_pages and itsi_service_analyzer.

For example:

$ curl -u admin:Splunk3r -k https://127.0.0.1:8089/servicesNS/nobody/SA-UserAccess/storage/collections/data/app_acl -X POST -H "Content-Type:application/json" -d '\{
"obj_id": "XXX-XXX-XXX",
"obj_type": "glass_table",
"obj_app": "itsi",
"obj_storename": "itsi_pages",

"obj_acl": \{
"obj_owner": "nobody",
"read": ["*"],
"write": ["*"],
"delete": ["*"]

},
"object_shared_by_inclusion": "true",
"acl_owner": "nobody"
}'
 

Bulk Import

Date filed Issue number Description
2019-09-17 ITSI-4402 Scheduled entity import isn't importing new entities.

Workaround:
  1. Open $SPLUNK_HOME/etc/apps/SA-ITOA/local/inputs.conf.
  2. In the itsi_csv_import stanza, make sure the entity_merge_field setting is set to a blank value. Remove the word undefined if it's there.

Deep Dive

Date filed Issue number Description
2019-05-22 ITSI-3258 "HTTP 414: URI Too Long" when navigating in the ITSI UI.

Workaround:
ITSI does not limit URL length, so pages with too many characters fail to load. To work around this issue, limit your request lengths to the following:
  • Browser request: < 2048 characters
  • REST request: < 8192 characters.

2018-09-13 ITSI-1556 When you drill down to a deep dive from the Predictive Analytics dashboard in Internet Explorer, the deep dive opens with no lanes because the URL is too long.

Workaround:
Manually add the KPI lanes to the deep dive.
2016-12-14 ITSI-525 If you zoom in on a specific time range in a deep dive while using twin-lane comparison, the comparisons that appear are occasionally offset by up to a minute.

Entities

Date filed Issue number Description
2019-10-28 ITSI-4721 Commas in the value of an entity's alias create duplicate entity aliases.
2019-06-27 ITSI-3654 Identical drilldown URLs to the entity details page are generated for entities with the same name.

Workaround:
Rename one of the VMs with the duplicate name
2019-02-19 ITSI-2540 The curl command to delete all entities times out with a large amount of entities.
2015-02-12 ITSI-1286 When importing entities using Data inputs > IT Service Intelligence CSV Import, the page overflows.

Entity Rules

Date filed Issue number Description
2019-06-19 ITSI-3534 Some entities are not clickable on the Service Analyzer.

Workaround:
This issue was fixed in version 4.3.0. To resolve the fix, upgrade to version 4.3.0 or later.
2019-06-10 ITSI-3443 Base searches do not properly associate entities based on the entity rules of a KPI's service.

Workaround:
This issue occurs due to an uppercase entity split field (for example, APPSERV). As a workaround, add an identical lowercase alias (for example, appserv) to each entity.

Notable Events

Date filed Issue number Description
2019-11-20 ITSI-4940 Nothing blocks you from creating an external ticket from an episode for which a ticket was already created.
2019-09-16 ITSI-4387 Closing an episode through the REST API does not break the episode, so it continues to receive events.

Workaround:
Generate a notable event that triggers a breaking condition in the aggregation policy, either from an actual events or from a notable event created directly from HEC.
2019-06-13 ITSI-3483, ITSI-3382 When using the "Link Ticket" option in Episode Review, the URL redirects to the wrong page.

Workaround:
Make sure the URL starts with http:// or https://. Otherwise the URL is interpreted as a relative URI.
2019-06-11 ITSI-3452 Upon upgrade to 4.2.x in a search head cluster, the event grouping custom command "itsirulesengine" may fail to run on some search heads: "ERROR Unable to invoke factory method in class class org.apache.logging.log4j.core.config.PropertiesPlugin".

Workaround:
To validate the root cause, log in to each search head and run the following search: | itsirulesengine
If the search fails on a search head, an error message appears in the UI and in the search.log. 
Once you have identified the offending search head, perform one of the following actions:
1. SSH to the search head and remove the following files:
cd /opt/splunk/etc/apps/SA-ITOA/lib/java/event_management/libs
 rm akka-actor_2.11-2.3.15.jar akka-slf4j_2.11-2.3.15.jar config-1.2.1.jar log4j-api-2.3.jar log4j-core-2.3.jar log4j-slf4j-impl-2.3.jar scala-library-2.11.5.jar slf4j-api-1.7.21.jar
Then retry the search.
2. If the files were pushed from the deployer, go to the deployer and remove the files:
cd /opt/splunk/shcluster/apps/SA-ITOA/lib/java/event_management/libs
 rm akka-actor_2.11-2.3.15.jar akka-slf4j_2.11-2.3.15.jar config-1.2.1.jar log4j-api-2.3.jar log4j-core-2.3.jar log4j-slf4j-impl-2.3.jar scala-library-2.11.5.jar slf4j-api-1.7.21.jar 
Then push the bundle to the search head and retry the search.
2019-05-14 ITSI-3185 Notable event archiving doesn't trigger because the itsi_notable_event_archive modular input is missing specification settings.

Workaround:
  1. Open $SPLUNK_HOME/etc/apps/SA-ITOA/README/inputs.conf.spec.
  2. In the stanza [itsi_notable_event_archive://<name>], add the following parameter:
    owner = <string>
  3. To confirm that the modular input works, check in index=_internal for events like "INFO ExecProcessor - New scheduled exec process: python /opt/splunk/etc/apps/SA-ITOA/bin/itsi_notable_event_archive.py".

2019-04-09 ITSI-2916 Episode Review displays "NaN" values in the event count column.

Workaround:
Either refresh the browser or refresh the Episode Review dashboard.
2019-04-04 ITSI-2904 Impacted services and KPIs don't show up and a console error occurs when loading Episode Review in real-time.
2019-02-15 ITSI-2532 Notable event aggregation policies occasionally don't pass tokens to actions.
2019-02-07 ITSI-2431 Episode Review does not generate events if there is no user with the username "admin" in Splunk, but owner=admin exists in the  stanza of etc/apps/SA-ITOA/metadata/default.meta.

Workaround:
Create a user with the username "admin" with the admin_all_objects capability and the itoa_admin role.
2019-01-03 ITSI-2164 ITSI backup times out due to an extremely large number of episode comments in the KV store.

Workaround:
Delete all comments prior to the backup (purge the collections in the KV store) or increase the Splunkd timeout and KV store limits. Then reduce the lifetime of the ITSI notable event collections in the KV store to archive them faster (the default is 6 months).
2018-12-10 ITSI-2059 Some notable events are added to more than one episode.

Workaround:
For an ITSI search head running Splunk 7.1 or 7.2, create or edit etc/system/local/limits.conf and add the following stanza: 
[search]
 phased_execution_mode = auto
 

For an ITSI search head running Splunk 7.3 or later, there is no need to change anything. 

2017-03-29 ITSI-1299 When your browser and the Splunk server are set to different DST time zones, the incorrect time might display for events in Episode Review.

Workaround:
Set your time zone to something other than "system default" even if you are in the same time zone as the system default.
2017-03-29 ITSI-1316 Splunkd connection fails due to "no_shared cipher matched" between client and server.

Workaround:
In order for notable event management and anomaly detection to work with Splunk platform 6.6, do the following:
  • Java 8/JRE 1.8/JDK 1.8*
* Download JCE 8 from http://www.oracle.com/technetwork/java/javase/downloads/jce8-download-2133166.html
* Unzip the downloaded file
* Place the two jars from the zip file into <java_jre_install_dir>/lib/security/ if running the JRE or <java_jdk_install_dir>/jre/lib/security if running the JDK.
  • Java 7/JRE 1.7/JDK 1.7*
* Download JCE 7 from http://www.oracle.com/technetwork/java/javase/downloads/jce-7-download-432124.html
* Unzip the downloaded file
* Place the two jars from the zip file into <java_jre_install_dir>/lib/security/ if running the JRE or <java_jdk_install_dir>/jre/lib/security if running the JDK.

Update SA-ITOA/local/commands.conf with the following commands: 

[itsirulesengine]

type = custom
command.arg.1=-J-Xmx1024M
command.arg.2=-Dlog4j.configurationFile=../default/log4j_rules_engine.xml
command.arg.3=-DitsiRulesEngine.configurationFile=../default/itsi_rules_engine.properties
command.arg.4=-Dhttps.protocols=TLSv1.2,TLSv1.1
command.arg.5=-Dhttps.cipherSuites=TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256,TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384,TLS_RSA_WITH_AES_128_CBC_SHA256
chunked = true

[itsicorrelationengine]

type = custom
command.arg.1=-J-Xmx1024M
command.arg.2=-Dlog4j.configurationFile=../default/log4j_correlation_engine.xml
command.arg.3=-J-XX:+UseConcMarkSweepGC
command.arg.4=-DitsiCorrelationEngine.configurationFile=../default/itsi_correlation_engine.properties
command.arg.5=-Dhttps.protocols=TLSv1.2,TLSv1.1
command.arg.6=-Dhttps.cipherSuites=TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256,TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384,TLS_RSA_WITH_AES_128_CBC_SHA256
chunked = true

Update SA-ITSI-MetricAD/local/commands.conf with the following commands:   

[mad]

type = custom
command.arg.1=-J-Xmx1G
command.arg.2=-Dlog4j.configurationFile=../default/log4j.xml
command.arg.3=-Dlog4j2.threadContextMap=com.splunk.mad.util.MadThreadContextMapcommand.arg.4=-Dhttps.protocols=TLSv1.2,TLSv1.1
command.arg.5=-Dhttps.cipherSuites=TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256,TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384,TLS_RSA_WITH_AES_128_CBC_SHA256chunked = true
2016-09-08 ITSI-1268 ITSI generates duplicate event_ids from the itsi_tracked_alerts index. This occurs when correlation search results contain an existing event_id. In this case, ITSI picks up the value of the event_id field and does not create a GUID for the event.

Workaround:
Rename the event_id field.
2016-04-01 ITSI-1346 The 'Ping Host' action does not work when ITSI and Enterprise Security are installed on the same machine.

Workaround:
1. Add the following stanza to $SPLUNK_HOME/etc/apps/SplunkEnterpriseSecurity/local/inputs.conf:
 [app_imports_update://update_es]
 apps_to_update = (SA-(?!(ITOA|ITSI|IndexCreation|UserAccess)).*) | (Splunk_SA_.*)


2. Delete the "import = *" line from [] stanza of $SPLUNK_HOME/etc/apps/$APP/metadata/local.meta, where APP=SA-ITOA, SA-ITSI-ATAD, SA-ITSI-LicenseChecker, SA-IndexCreation, SA-UserAccess.
3. Restart Splunk.

Glass Table

Date filed Issue number Description
2019-10-09 ITSI-4617 KPI and ad hoc searches in beta glass tables are run under the context "App: Search" instead of "App: ITSI". This prevents certain app-restricted searches such as Predictive Analytics searches from running.
2019-06-13 ITSI-3470 Geometric shapes (rectangles, circles) with static color lose their color after upgrade to the beta framework.
2019-05-29 ITSI-3317 Using "Save As" for a classic glass table makes it disappear from the lister page.
2019-05-28 ITSI-3312 If the Width or Heigh field of a KPI widget in a classic glass table contains a decimal, the glass table cannot be upgraded to the beta framework.
2019-05-24 ITSI-3291 Custom icons uploaded to a classic glass table that is then upgraded to the beta framework cannot be backed up.
2019-05-03 ITSI-3095 Visualizations with keys that contains dots "." in the source editor (for example, "axisTitleY.text") cannot be saved.
2019-04-12 ITSI-2951 The 'Edit KPI' button in the beta glass table widget configuration leads to the wrong page.
2019-04-05 ITSI-2906 Some beta glass table widgets aren't displaying data.
2018-09-14 ITSI-1567 When you add a predictive model to a glass table, you cannot use the sparkline or trending value viz types because the prediction is a static value.

KPI Base Searches

Date filed Issue number Description
2019-09-06 ITSI-4275 Multiple KPIs that are linked to shared base searches fail to populate.

Workaround:
Increase timeouts in ITSI commands.
  1. Add the following stanza to $SPLUNK_HOME/etc/apps/SA-ITOA/local/itsi_settings.conf:
    [customsearch]
    
    timeout_read = 360 
  2. Add the following setting to $SPLUNK_HOME/etc/apps/SA-ITOA/lib/ITOA/storage/statestore.py:
    REST_TIMEOUT = 6000

2019-06-10 ITSI-3443 Base searches do not properly associate entities based on the entity rules of a KPI's service.

Workaround:
This issue occurs due to an uppercase entity split field (for example, APPSERV). As a workaround, add an identical lowercase alias (for example, appserv) to each entity.

KPI Search Calculation

Date filed Issue number Description
2019-04-29 ITSI-3045, SPL-163319 Despite the forceCsvResults parameter not existing in the configuration for a saved search with summary indexing enabled, the summarized data is improperly populated with this parameter.

Workaround:
Add the following stanza to $SPLUNK_HOME/etc/apps/itsi/local/alert_actions.conf:
[indicator]
command = rename _raw as orig_raw | eval qf=if(alert_level==-2,"maintenancerandostring","") |  eval itsi_service_id=if(isnull(itsi_service_id) AND isnotnull("$action.indicator._itsi_service_id$") AND trim("$action.indicator._itsi_service_id$")!="","$action.indicator._itsi_service_id$",itsi_service_id) | eval itsi_kpi_id=if(isnull(itsi_kpi_id) AND isnotnull("$action.indicator._itsi_kpi_id$") AND trim("$action.indicator._itsi_kpi_id$")!="","$action.indicator._itsi_kpi_id$",itsi_kpi_id) | summaryindex spool=t uselb=t addtime=t index="$action.indicator._name{required=yes}$" file="$name_hash$_$#random$.stash_new" name="$name$" marker="$action.indicator*{format=$KEY=\\\"$VAL\\\", key_regex="action.indicator.(?!(?:command|inline|forceCsvResults|maxresults|maxtime|ttl|track_alert|(?:_.*))$)(.*)"}$"

2019-04-22 ITSI-3006 When creating a KPI that uses metrics data, the KPI editor takes a long time to load the list of available metrics.

Maintenance Window

Date filed Issue number Description
2018-04-25 ITSI-277, ITSI-3448 The maintenance window UI does not calculate daylight savings correctly.

Workaround:
The maintenance window UI displays the UTC time in parentheses. Rely on these times for the maintenance boundaries.

Performance

Date filed Issue number Description
2019-05-07 ITSI-3115 Upon upgrade from 4.0.3 to 4.1.x or 4.2.x, changing a KPI causes CPU usage to spike and an update delay.

Workaround:
Manually disable backfill on all affected KPIs.

Role Based Access Controls

Date filed Issue number Description
2019-03-29 ITSI-2860 If you assign the write_itsi_correlation_search capability to the itoa_analyst role, the role still cannot create a correlation search.

Workaround:
In addition to assigning the write_itsi_correlation_search capability to the itoa_analyst role, create a local.meta file at SPLUNK_HOME/etc/apps/itsi/metadata/ and add "itoa_analyst" to the [savedsearches] stanza.

For example:

[savedsearches]
access = read : [ * ], write: [ itoa_admin, itoa_team_admin, itoa_analyst ], delete: [ itoa_admin, itoa_team_admin, itoa_analyst ]
export = system

2018-02-06 ITSI-440 When itoa_admin, itoa_analyst, itoa_team_admin and itoa_user roles are added to a new custom role, users assigned to the custom role do not have the "edit permissions" capability for saved service analyzers.
2017-10-16 ITSI-437 Roles inheriting from itoa_admin do not behave like itoa_admin. For example, the inheriting role cannot edit permissions on pages such as glass tables, deep dives, and service analyzers.

Workaround:
Make the user a member of the itoa_admin role (rather than just a member of a role inheriting from it).

Service Analyzer

Date filed Issue number Description
2019-08-07 ITSI-3963 When you select a service tile in the Service Analyzer, the get_itsi_summary_index search either fails or executes an all-time search.
2019-06-19 ITSI-3534 Some entities are not clickable on the Service Analyzer.

Workaround:
This issue was fixed in version 4.3.0. To resolve the fix, upgrade to version 4.3.0 or later.
2019-05-22 ITSI-3258 "HTTP 414: URI Too Long" when navigating in the ITSI UI.

Workaround:
ITSI does not limit URL length, so pages with too many characters fail to load. To work around this issue, limit your request lengths to the following:
  • Browser request: < 2048 characters
  • REST request: < 8192 characters.

2019-02-21 ITSI-2562 Backend Service Analyzer searches ignore filtering and calculate statistics against all services, leading to significantly longer than expected search times.
2017-10-04 ITSI-1290 Filters with no matching results can't be saved in the Service Analyzer.

Service Definition

Date filed Issue number Description
2016-03-28 ITSI-1269 On Windows 10 on Chrome, some selectors in the ITSI app do not function.

Threshold Templates

Date filed Issue number Description
2019-04-08 ITSI-2914 When you first add a new KPI to a service template and apply Adaptive Thresholding, the additional KPI reuses the preview of the first KPI that was added to the template and displays misleading threshold values.

Workaround:
Once the scheduled daily adaptive threshold update runs, all KPIs linked to the template are correctly updated. Wait until midnight for the adaptive threshold values to update themselves.
2018-12-05 ITSI-2020 When you run the kvstore_to_json.py mode 3 option on ALL KPI threshold templates (versus just one), the KPI does not reflect the changes made.

Predictive Analytics

Date filed Issue number Description
2019-10-09 ITSI-4617 KPI and ad hoc searches in beta glass tables are run under the context "App: Search" instead of "App: ITSI". This prevents certain app-restricted searches such as Predictive Analytics searches from running.
2019-10-01 ITSI-4530, ITSI-4604 The KPI Predictions chart on the Predictive Analytics dashboard does not display the correct timestamps.
2019-10-01 ITSI-4531 The Predictive Analytics Dashboard "KPI Predictions" panel plots results in GMT rather than the user's timezone.
2019-03-20 ITSI-2801 Predictive Analytics occasionally fails to train models on Windows.

Workaround:
If search.log for the fit command reports the following error:

ERROR ChunkedExternProcessor - stderr: ImportError: DLL load failed: The application has failed to start because its side-by-side configuration is incorrect. Please see the application event log or use the command-line sxstrace.exe tool for more detail.

Then reinstall of Visual C++ 2008 runtime to resolve the issue: https://www.microsoft.com/en-in/download/details.aspx?id=40784

2019-01-18 ITSI-2309 Predictive Analytics is not available if ITSI is installed on Splunk Enterprise version 7.0.x.

Workaround:
Perform one of the following workarounds:
A. Upgrade to Splunk version 7.1.x or later.

B. If you cannot upgrade, modify the Predictive Analytics macros:

1. Navigate to $SPLUNK_HOME/etc/apps/SA-ITOA/local
2. Create or edit a macros.conf file.
3. Add the following stanza to the file:
# Macro to train KPI trend models and health score KPI relations.
[train_kpi_trends(2)]
args=sid,suffix
definition = `itsi_predictive_analytics_dataset($sid$)`\
  | appendpipe [fit LinearRegression fit_intercept=true now_avg_hs from\
    "value_avg:*" into app:itsi_predict_kpi_hs_$suffix$ | fields - _time *]\
  | fit StandardScaler "value_*" with_mean=true with_std=true into app:itsi_predict_kpi_ss_$suffix$\
  | `prepare_kpi_trend_data($sid$,$suffix$)`\
  | map search="| inputcsv itsi_predict_kpi_$suffix$.csv | fit GradientBoostingRegressor \"next30mkpi_$kpiid$\" from\
    \"SS_*\" \"this_date_*\" \"last30mkpi_$kpiid$\" \"value_avg: $kpiid$\" into app:itsi_predict_kpi_$model_suffix$"\
    maxsearches=100\
  | head 1\
  | fields "predicted(*)"\
  | rename "predicted(next30mkpi_*)" as *\
  | fields - _time\
  | foreach * [eval <<FIELD>>=1]\
  | untable modelname kpi dummyfield\
  | fields - dummyfield\
  | eval modelname="itsi_predict_kpi_".replace(kpi, "-", "_")\
  | append [| listmodels\
    | search name="itsi_predict_kpi_*_$suffix$"\
    | rename name as modelname\
    | fields modelname]

4. Save the file and restart Splunk.

5. Verify the fix by training a predictive model for a small time period (like 7 days).
2018-09-14 ITSI-1567 When you add a predictive model to a glass table, you cannot use the sparkline or trending value viz types because the prediction is a static value.
2018-09-13 ITSI-1556 When you drill down to a deep dive from the Predictive Analytics dashboard in Internet Explorer, the deep dive opens with no lanes because the URL is too long.

Workaround:
Manually add the KPI lanes to the deep dive.
2018-08-01 ITSI-1105 After you delete a Predictive Analytics model through Lookups, the model still appears in the UI.

Splunk App for Infrastructure Integration

Date filed Issue number Description
2019-06-17 ITSI-3512 It is possible to edit the default SAI service templates shipped with ITSI, but all changes are overwritten upon upgrade.
2019-04-22 ITSI-3006 When creating a KPI that uses metrics data, the KPI editor takes a long time to load the list of available metrics.
2019-04-21 ITSI-2996 SAI group alerts without an entity identifier do not have a working drilldown link. The link goes to a blank workspace.
2018-09-24 ITSI-1654 Only 50,000 entities can be imported from the Splunk App for Infrastructure.

Workaround:
By default, the entity integration imports up to 50,000 entities from the Splunk App for Infrastructure. If you have more than 50,000 entities in Splunk App for Infrastructure, only the first 50,000 will be imported into ITSI. Increase the max_rows_per_query setting in $SPLUNK_HOME/etc/apps/SA-ITOA/local/limits.conf under the [kvstore] stanza to import more than 50,000 entities.

Uncategorized issues

Date filed Issue number Description
2019-08-23 ITSI-4171 When your system's time zone and the Splunk time zone set in your user preferences are different, it may cause several hours of lag between Rules Engine logs and Python logs in the _internal index.

Workaround:
Configure your Splunk time zone to be the same as your system's time zone.
2019-08-05 ITSI-3924 An error in the multi-KPI "status over time" alert search results in the percentages always been 100%.

Workaround:
This error occurs because the stats count is creating the field occurrences, but the getPercentage macro is expecting the field occurrence. To work around this issue, open the multi-KPI alert in the correlation search editor and change the word "occurrence" to "occurrences". Note that this action prevents you from using the Multi-KPI Alerts page to edit the correlation search in the future.
2019-07-24 ITSI-3836 Objects such as service analyzers, glass tables, and deep dives are missing after upgrade.

Workaround:
If some objects are missing from the UI or unaccessible after you upgrade, the ACL objects corresponding to the objects might be missing or corrupted. For troubleshooting steps, see https://docs.splunk.com/Documentation/ITSI/latest/Install/Troubleshoot.
2019-07-01 ITSI-3666 Upon upgrade, the Splunk product name changes from Splunk>enterprise to Splunk>hunk.
2019-02-12 ITSI-2471 If ITSI is installed on multiple environments with multiple license masters, and any indexer interacts with both environments, a duplicate licensing error occurs because both environments have the same auto-generated ITSI license stack.

Workaround:
Delete the internal license, install a secondary internal license, and disable the license_checker modular inputs.

1. Click Settings > Licensing and delete the IT Service Intelligence Internals *DO NOT COPY* stack.

2. Click Add license and upload the following license key file:

<license>
 <signature>o3eXzWryQOQG3M2d1vs9dSn8NsxXbB1HtozqcaTkjo9QhHzZTLFWup1zakfCbnYJusY+WmT2EepcjdD7e2QSlvgGCPocOehXBmiXDWkfOGboMLQ8PYOTV/W+hdZUkO+hakllZuVKdAwUlnBQfJPIS4EgYLtFd+BwmpIK887PtgB7Tohm4jaeEdnkJpsKZSymv/ZPVs6aD+6PLvSi2WLEo5mwyvRz12PqiQiwTLkfT1FVXUe70rZGsJM3udiGuB4KLuNmvdSK5W/c7uMvWOTys6jY4XIPn7dlLWick4SQ9zqm3WlwIeJOCfD3J8u7kKjo5kdzuAxc3Qr3C0j/c4yRUA==</signature>
 <payload>
 <type>fixed-sourcetype</type>
 <group_id>Enterprise</group_id>
 <quota>107374182400000</quota>
 <max_violations>5</max_violations>
 <window_period>30</window_period>
 <creation_time>1549958400</creation_time>
 <label>IT Service Intelligence Internals *DO NOT COPY*</label>
 <expiration_time>2163135600</expiration_time>
 <features>
 <feature>Auth</feature>
 <feature>FwdData</feature>
 <feature>LocalSearch</feature>
 <feature>ScheduledSearch</feature>
 <feature>Alerting</feature>
 <feature>SplunkWeb</feature>
 </features>
 <add_ons>
 <add_on name="itsi" type="app">
 <parameter key="size" value="1"/>
 </add_on>
 </add_ons>
 <sourcetypes>
 <sourcetype>itsi_notable:*</sourcetype>
 </sourcetypes>
 <guid>71029F93-1CBD-4201-8D8D-03D0EAD582A0</guid>
 </payload>
 </license>
 

3. Click Settings > Data inputs > IT Service Intelligence license checker and disable both inputs.

2018-11-16 ITSI-1941 When you create a multi-KPI alert, the summary index stores the entity_title as the search head and not the entity used to populate the data.

Workaround:
Create a correlation search as an alternative to a multi-KPI alert.

1. Click Configure > Correlation Searches.

2. Click Create New Search > Create Correlation Search.

3. Provide a search name.

4. Enter a search that contains the service ID. For example, `mka_sn_kpin("Password Reset Tool","CPU Utilization: %")`.

5. Enter a notable event title and description. For example, %service_name% degraded because of %entity_title%.

6. Configure other fields and click Save to save the correlation search.

7. Go to Episode Review and you should start seeing events.

2018-06-27 ITSI-1287, ITSI-793 Correlation searches created by manually editing savedsearches.conf do not appear on the correlation search lister page.

Workaround:
Do not create correlation searches by manually editing $SPLUNK_HOME/etc/apps/itsi/local/savedsearches.conf. The search will not appear on the correlation search lister page. Always create correlation searches directly in the IT Service Intelligence app.
2015-12-01 ITSI-1320 When you install Enterprise Security on a search head with a pre-existing installation of ITSI, the ES-specific roles overwrite the ITSI-specific roles assigned to admin role. This disables access to all read/write objects in ITSI.

Workaround:
1. In Splunk Web, go to Settings > Access Controls.

2. Select Roles > admin.
3. Add itoa_admin, itoa_analyst, and itoa_user to Selected roles.
4. Click Save.

2015-03-25 ITSI-1293 In a search head cluster environment, you cannot set up a recurring import (from CSV or search) through the UI.

Workaround:
1. Create the modular input through the UI. ITSI adds the input as a new stanza in $SPLUNK_HOME/etc/apps/itsi/local/inputs.conf. It is not replicated across search peers.

Alternatively, if you're familiar with the format of modular inputs, you can create the input yourself.
2.Copy the input stanza from the local version of inputs.conf and add it to shcluster/apps/itsi/local/inputs.conf on the deployer.
3. Let the deployer push the file to the search peers. The file is deployed to the default inputs.conf on each search peer.
4. Remove the modular input stanza from $SPLUNK_HOME/etc/apps/itsi/local/inputs.conf on the search head that created it. Otherwise it will take precedence on the deployer.

All ITSI Modules

Publication date Issue number Description
2017-03-21 ITOA-7585 When you bulk add services and an error caused by the racing condition occurs, the incorrect message "itsi_module does not exist" is displayed.
2017-03-07 MOD-979 KPIs do not have consistent backfill settings across all modules.
2017-01-17 MOD-452 The Analyze KPI button on the Service Details page is broken.
2017-01-17 MOD-402 The Export to PDF option does not work in the drilldown to a module.
2017-01-17 MOD-296 The extendable tab XML generator REST endpoint is located in DA-ITSI-OS instead of in common components where it can be used by all modules.
2017-01-17 MOD-591 ITSI displays a misleading error message when a KPI template contains a field that cannot be resolved.
2017-01-17 MOD-498 There is no upper limit to the number of characters a KPI title or description can contain. Long strings can negatively affect performance.
2017-01-17 MOD-309 The Gruntfile.js included in ITSI modules uses double quotes instead of single quotes, which does not conform to the standard for all JavaScript files.
2017-04-17 MOD-2002 When you drilldown from the Events tab, an "Invalid earliest_time" error occurs.


Workaround:
Disable drilldown from the Events tab.

2017-01-17 MOD-439 Some modules do not have descriptions for saved searches.

Application Server Module

Publication date Issue number Description
2017-01-27 MOD-492 If you reuse the same panel within a dashboard, the duplicate panel does not display any event data.

Cloud Services Module

There are no known issues for this release.

Database Module

Publication date Issue number Description
2017-01-17 MOD-586 When a lookup is not configured for TA-Microsoft-SqlServer, ITSI displays a misleading error message on the server drilldown page.

End User Experience Module

There are no known issues for this release.

Load Balancer Module

Publication date Issue number Description
2017-01-27 MOD-492 If you reuse the same panel within a dashboard, the duplicate panel does not display any event data.

Operating System Module

Publication date Issue number Description
2017-04-13 MOD-555 The Storage Free Space % base search runs every minute while the Linux df command runs every 5 minutes. This causes data gaps.
2017-04-10 MOD-1964 Windows data for memory free space is collected at different intervals than the Memory Free % KPI.
2017-01-17 MOD-1398 Line, stack, and area charts do not display a metric gap when no metrics are available during a time period.

Storage Module

There are no known issues for this release.

Virtualization Module

There are no known issues for this release.

Web Server Module

Publication date Issue number Description
2017-03-17 MOD-320 Some KPI ad hoc searches transform data with the stats command and do not retain time fields. The KPIs do not render anything and do not show thresholding details.
2017-03-17 MOD-538 When you add a new tab with panels and refresh the page, the page breaks.
PREVIOUS
Fixed issues in Splunk IT Service Intelligence
  NEXT
Removed features in Splunk IT Service Intelligence

This documentation applies to the following versions of Splunk® IT Service Intelligence: 4.2.1


Was this documentation topic helpful?

Enter your email address, and someone from the documentation team will respond to you:

Please provide your comments here. Ask a question or make a suggestion.

You must be logged into splunk.com in order to post comments. Log in now.

Please try to keep this discussion focused on the content covered in this documentation topic. If you have a more general question about Splunk functionality or are experiencing a difficulty with Splunk, consider posting a question to Splunkbase Answers.

0 out of 1000 Characters