Splunk® IT Service Intelligence

Release Notes

Acrobat logo Download manual as PDF


Splunk IT Service Intelligence (ITSI) version 4.9.x will reach its End of Life on April 21, 2023. See the Splunk Software Support Policy for details. For information about upgrading to a supported version, see Before you upgrade IT Service Intelligence.
This documentation does not apply to the most recent version of Splunk® IT Service Intelligence. Click here for the latest version.
Acrobat logo Download topic as PDF

Known issues in Splunk IT Service Intelligence

IT Service Intelligence (ITSI) version 4.9.2 has the following known issues and workarounds.

Adaptive Thresholding

Date filed Issue number Description
2021-11-05 ITSI-19663 Updating a KPI threshold policy within a service template causes the thresholds of all existing KPIs that use adaptive thresholds to get reset

Workaround:
Temporary workaround to avoid false alerts: 
# Put services that are linked to the service template into maintenance mode
# Make KPI threshold changes within the service template and push out
# Wait to make sure all services are synced
# Manually run the itsi_at_search_kpi_minusXd to recreate the adaptive threshold models
# Disable maintenance mode for false alerts
2021-07-28 ITSI-17991 AdaptiveThresholding: KPI calculated thresholdValues get overwritten

Bulk Import

Date filed Issue number Description
2021-07-03 ITSI-17629, ITSI-17628 Entity Import: Recurring import saved with "role" instead of user name for 'Owner'
2021-06-09 ITSI-17178 Some ITSI Import Objects saved searches fail to merge entities with the host field and may create duplicate entities.

Workaround:
#Disable ITSI Import Objects - VMware VM.
  1. Copy the ITSI Import Objects - VMware VM saved search, but change the entity_merge_field attribute to host.
  1. Enable the updated ITSI Import Objects - VMware VM search.

Entity Rules

Date filed Issue number Description
2021-10-18 ITSI-19304 Newly created entities are not reflected on the service analyzer

Notable Events

Date filed Issue number Description
2021-12-07 ITSI-20343 Impacted Services and KPIs do not appear in Episode Review when using Teams functionality

Workaround:
Create/edit Template:SA-ITOA/local/macros.conf and add the following two stanzas:

{noformat}# Take in a string of team_keys in the format of '(sec_grp="itsi_team_key") OR (sec_grp="itsi_team_key")' and returns a filter of service_ids [itsi_events_compare_teams(1)] args = itsi_team_id_list definition = search (service_ids=*null*) OR (NOT service_ids=*) OR [|inputlookup itsi_services_in_team_lookup where ($itsi_team_id_list$) | rename _key as service_ids | eval service_ids="*".service_ids."*" | fields service_ids]

  1. Take in a string of team_keys in the format of '(sec_grp="itsi_team_key") OR (sec_grp="itsi_team_key")' and returns a filter of itsi_service_ids

[itsi_groups_compare_teams(1)] args = itsi_team_id_list definition = search (itsi_service_ids=*null*) OR (NOT itsi_service_ids=*) OR [|inputlookup itsi_services_in_team_lookup where ($itsi_team_id_list$) | rename _key as itsi_service_ids | eval itsi_service_ids="*".itsi_service_ids."*" | fields itsi_service_ids]{noformat}

2021-10-20 ITSI-19415 On Windows server, more than 1 rules engines processes are spawned at a time.

Workaround:
The root cause is the splunk phased_execution_mode. Edit the limits.conf file and add the line:  

[search]
phased_execution_mode = auto
2021-09-16 ITSI-18910 For fresh ITSI install, populate installed_version and old_version values
2021-07-15 ITSI-17809 After ITSI upgrade from 4.6.1 to 4.9.1, new episodes are created instead of events being added to existing episodes.

Workaround:
Add the policy_id in the field_list for the itsi_notable_group_system_lookup stanza in $SPLUNK_HOME/etc/apps/SA-ITOA/transforms.conf
2021-05-24 ITSI-16844 For time-based policies, Action Rules with the condition AND "the following event occurs" don't work.
2021-01-21 ITSI-13167 On Safari, there is a 10 to 15 second delay when editing a Notable Event Aggregation Policy using the ServiceNow action

Notable Event Aggregation Policies

Date filed Issue number Description
2021-12-07 ITSI-20343 Impacted Services and KPIs do not appear in Episode Review when using Teams functionality

Workaround:
Create/edit Template:SA-ITOA/local/macros.conf and add the following two stanzas:

{noformat}# Take in a string of team_keys in the format of '(sec_grp="itsi_team_key") OR (sec_grp="itsi_team_key")' and returns a filter of service_ids [itsi_events_compare_teams(1)] args = itsi_team_id_list definition = search (service_ids=*null*) OR (NOT service_ids=*) OR [|inputlookup itsi_services_in_team_lookup where ($itsi_team_id_list$) | rename _key as service_ids | eval service_ids="*".service_ids."*" | fields service_ids]

  1. Take in a string of team_keys in the format of '(sec_grp="itsi_team_key") OR (sec_grp="itsi_team_key")' and returns a filter of itsi_service_ids

[itsi_groups_compare_teams(1)] args = itsi_team_id_list definition = search (itsi_service_ids=*null*) OR (NOT itsi_service_ids=*) OR [|inputlookup itsi_services_in_team_lookup where ($itsi_team_id_list$) | rename _key as itsi_service_ids | eval itsi_service_ids="*".itsi_service_ids."*" | fields itsi_service_ids]{noformat}

2021-10-20 ITSI-19415 On Windows server, more than 1 rules engines processes are spawned at a time.

Workaround:
The root cause is the splunk phased_execution_mode. Edit the limits.conf file and add the line:  

[search]
phased_execution_mode = auto
2021-09-16 ITSI-18910 For fresh ITSI install, populate installed_version and old_version values
2021-07-15 ITSI-17809 After ITSI upgrade from 4.6.1 to 4.9.1, new episodes are created instead of events being added to existing episodes.

Workaround:
Add the policy_id in the field_list for the itsi_notable_group_system_lookup stanza in $SPLUNK_HOME/etc/apps/SA-ITOA/transforms.conf
2021-05-24 ITSI-16844 For time-based policies, Action Rules with the condition AND "the following event occurs" don't work.
2021-01-21 ITSI-13167 On Safari, there is a 10 to 15 second delay when editing a Notable Event Aggregation Policy using the ServiceNow action

Glass Table

Date filed Issue number Description
2021-08-13 ITSI-18306 "Service not found" error when switching services in Glass Tables if the service isn't included in the first 100 services

Workaround:
the only workaround I can think off (which I understand is not ideal) would be to rename the services that user want to user for service swapping of glass table with a prefix like A or something at the beginning of the alphabet or 0 if the service names have numbers. The idea is to have the services display in the first page of the service lister when the count per page is set to 100. Or within the first 5 pages if the count per page is set to 20.

KPI Base Searches

Date filed Issue number Description
2021-07-28 ITSI-17989 KPI Base Search name is limited to about 70 characters.

KPI Search Calculation

Date filed Issue number Description
2022-01-10 ITSI-21013 With custom indexes, when creating new KPI, the backfill checks look to the default itsi_summary instead of the custom one, causing potentially extra backfill.

Maintenance Window

Date filed Issue number Description
2021-03-04 ITSI-14296 Maintenance Window time validation control errors : custom time set throws the error, "Start/End time date is invalid"

Workaround:
After setting the Start Time to what you want, change the start date to another date, and then change it back to the date that you want. Doing this clears the error, and you can proceed by clicking <bold>Next</bold>.

Service Analyzer

Date filed Issue number Description
2021-10-18 ITSI-19304 Newly created entities are not reflected on the service analyzer
2021-08-05 ITSI-18101 Service Analyzer Entity Panel always shows N/A after selecting a KPI that is N/A

Service Templates

Date filed Issue number Description
2021-11-24 ITSI-20208 Service Template and Service stuck in Syncing status

Workaround:
Following the steps in the below doc, we identified and generated output that shows the Service Templates with missing services.

Eng team in IST timezone will update the migration.zip for Support to run the fix. 

[1]

Uncategorized issues

Date filed Issue number Description
2023-01-09 ITSI-27961 Bidirectional Ticketing Correlation Search hits "subsearch limit of 50000 reached" when the collection itsi_notable_event_ticketing has more than 50000 entries

Workaround:
# Navigate to ITSI -> Configuration -> Correlation Searches
  1. Click on Bidirectional Ticketing
  2. Paste the following search in the Search field and then click on Save. Also enable the CS if it has been disabled

{noformat}| datamodel Ticket_Management Incident search | rename All_Ticket_Management.ticket_id as ticket_id | join ticket_id [search sourcetype="snow:incident" index="<snow_index>" | where _indextime > now() - <max_lookback_time>] | lookup itsi_notable_event_external_ticket tickets.ticket_id as ticket_id OUTPUTNEW tickets.ticket_system event_id | where isnotnull(event_id) | rename tickets.* as * | eventstats values(event_id) as group_id last(ticket_system) as ticket_system by ticket_id | fields - dv_* | table * | makemv group_id | mvexpand group_id | eval bidirectional_ticketing=1, snow_hash = number + "!" + group_id + "!" + sys_updated_on | search NOT [| search index="itsi_tracked_alerts" | fields snow_hash] | dedup snow_hash{noformat}

Change the placeholders {{<snow_index>}} and {{<max_lookback_time>}} in the above search with values according to the customer's requirements

2022-03-24 ITSI-22641 Premium features disabled because the ITSI license checker is not finding all the valid licenses, when they are more than 30 licenses installed

Workaround:
If the customer has more than 30 licenses, remove the expired ones to keep the list short.
2021-12-14 ITSI-20605, ITSI-22366 Occasionally after ITSI upgrade, non-admin users get Oops Page - local.meta corrupted during the upgrade

Workaround:
Clean up all permissions on ITSI views in itsi/metadata/local.meta (and sync on SHC)

The workaround is to clean up the stanza in local.meta on the all the SH. remove all the stanza like \[views/....]  that have no valid access settings, (access = delete : \[  ], read : \[  ], write : \[  ]) and that are not custom views from your users.

As they may be many, to confirm, you can compare to the list in default.meta And you also can look at the modtime field in the stanza, as they are probably all identical.

2021-11-27 ITSI-20219 Inefficient background searches initiated during ITSI KV store migration may cause SmartStore outage.

Workaround:

2021-10-06 ITSI-19153 Infrastructure Overview Centre Pane needs to be resizeable
2021-10-04 ITSI-19103, ITSI-19699 Data Integration UI is not showing all content pack chiclets
2021-09-22 ITSI-18967, ITSI-19697 Content Pack upgrade fails as already installed ITSI objects are not shown as "Already Present" in Content Pack UI during upgrade scenario for CP present in Content library
2021-09-09 ITSI-18800 When you add ITSI instances as search peers to another Splunk instance, the peers might be disabled after 72 hours. This is because the ITSI licenses are flagged as duplicates on the search peers.

Workaround:
#Go to the node search peer manager node.
  1. Identify the Splunk licenses (Enterprise, ITSI, non-ITSI) currently installed. Ignore licenses under the *IT Service Intelligence Internals DO NOT COPY* stack.
  1. Navigate to http://LM_IP/en-US/manager/system/licensing/licenses and check if the AllowDuplicateKeys capability is enabled for each of the license identified in step 1.
  1. If not enabled, procure a new license from Splunk support and replace it.
  1. Make sure all licenses in the stack have the capability enabled.
  1. Restart Splunk.
2021-09-01 ITSI-18709 ITSI redirects to suite_redirect 500 Internal Server Error - because of python library isolation between apps

Workaround:
Step 1: Identify all the splunklib directories within the splunk apps directory using command find . -name 'splunklib' | xargs -r ls -lah.

Step 2: For each directory listed in step 1, check if file six.py is present.

Step 3: Copy the six.py from an existing splunklib directory into all the missing directories.

Step 4: Clean the cached files using find . -name "*.pyc" -delete

Step 5: Reload the ITE Work app.

2021-08-17 ITSI-18390 Can't create metric-based KPIs for a service in ITSI 4.9.2

Workaround:
You must use an ad-hoc search using the | mstats command to create metrics based KPIs. 
2021-07-22 ITSI-17901 After installing IT Essentials Work, receive an error of ERROR ExecProcessor - message from ""C:\Program Files\Splunk\bin\Python3.exe"
2021-05-25 ITSI-16870, ITSI-17181 Entity page doesn't load when IT Essentials Work is installed on search head cluster
2021-05-12 ITSI-16577 SA-ITOA transforms refers to non existing collections - causing errors about MongoModificationsTracker in splunkd.log

Workaround:
The workaround is either to delete the following unused stanzas from SA-ITOA/default/transforms.conf or replace these unused stanzas in SA-ITOA/local/transforms.conf: 

[itsi_data_integrations_aws]
 collection = itsi_services

[itsi_data_integrations_azure]
 collection = itsi_services

[itsi_data_integrations_cloud_setup]
 collection = itsi_services

2021-03-24 ITSI-15026 Entities discovered before entity types are added may have no entity type association.

Workaround:
On a reinstallation of ITSI, don't enable entity discovery saved searches before pre-packaged entity types are added to ITSI. 
2019-05-30 ITSI-3322 If you add a correlation search in ITSI which contains a sub-search returning into an eval, you get a message "Invalid search string: This search cannot be parsed when parse_only is set to true."

Workaround:
You can't use a sub-search returning into an eval in a correlation search. As a workaround, create and save a basic correlation search with all of the information you want outside of the search. Then as an admin user, go to Settings > Searches, reports, and alerts and open the correlation search you just created. Add the sub-search you were trying to add there.
Last modified on 07 April, 2023
PREVIOUS
Fixed issues in Splunk IT Service Intelligence 		  NEXT
Removed features in Splunk IT Service Intelligence

This documentation applies to the following versions of Splunk® IT Service Intelligence: 4.9.2

Was this documentation topic helpful?


You must be logged into splunk.com in order to post comments. Log in now.

Please try to keep this discussion focused on the content covered in this documentation topic. If you have a more general question about Splunk functionality or are experiencing a difficulty with Splunk, consider posting a question to Splunkbase Answers.

0 out of 1000 Characters