Splunk® IT Service Intelligence

Release Notes

Acrobat logo Download manual as PDF


Splunk IT Service Intelligence (ITSI) version 4.11.x reached its End of Life on December 6, 2023. See the Splunk Software Support Policy for details. For information about upgrading to a supported version, see Before you upgrade IT Service Intelligence.
This documentation does not apply to the most recent version of Splunk® IT Service Intelligence. For documentation on the most recent version, go to the latest release.
Acrobat logo Download topic as PDF

Known issues in Splunk IT Service Intelligence

IT Service Intelligence (ITSI) version 4.11.2 has the following known issues and workarounds.

Adaptive Thresholding

Date filed Issue number Description
2021-11-05 ITSI-19663 Updating a KPI threshold policy within a service template causes the thresholds of all existing KPIs that use adaptive thresholds to get reset

Workaround:
Temporary workaround to avoid false alerts: 
# Put services that are linked to the service template into maintenance mode
# Make KPI threshold changes within the service template and push out
# Wait to make sure all services are synced
# Manually run the itsi_at_search_kpi_minusXd to recreate the adaptive threshold models
# Disable maintenance mode for false alerts

Backup/Restore and Migration Issues

Date filed Issue number Description
2021-12-06 ITSI-20325 When a backup .ZIP file includes a base search with a title that is over 100 characters, the backup restore job will fail.
2021-12-02 ITSI-20308 Errors found in the migration log while upgrading to 4.11.0
2021-10-13 ITSI-19215 Customer is getting a lot of errors related to "Could not find object id=itsi_entity_dashboard_drilldown" after installation of IT Essentials Work

Workaround:
Upgrade to ITE-Work version 4.12 and later

Notable Events

Date filed Issue number Description
2022-12-20 ITSI-27751 Episode Review arbitrary search filter with AND & OR conditions fail to match events under certain scenarios

Workaround:
Avoid using brackets (), extra whitespaces, the operator !=, and double quotes "" in the search filter
2022-01-25 ITSI-21269 The grouping of Bidirectional Ticketing events sets the episode KV store state to faulty values
2022-01-05 ITSI-20978 ServiceNow bidirectional integration resets the episode title, status, severity, and owner.

Workaround:
#In etc/app/itsi/local/savedsearches.conf, replace the Bidirectional Ticketing clause with the following:

\[Bidirectional Ticketing] action.itsi_event_generator.param.description = %group_description% action.itsi_event_generator.param.itsi_instruction = %group_instruction% action.itsi_event_generator.param.owner = %group_assignee% action.itsi_event_generator.param.severity = %group_severity% action.itsi_event_generator.param.status = %group_status% action.itsi_event_generator.param.title = %group_title% disabled = 0 dispatch.earliest_time = -4h search = | `itsi_bidirectional_ticketing(main,90,itsi_tracked_alerts)`\\ | join group_id\\     \[ search `itsi_event_management_group_index_with_state("")` | rename itsi_group_id as group_id | stats latest(itsi_group_title) as group_title latest(severity) as group_severity latest(owner) as group_assignee latest(itsi_group_description) as group_description latest(itsi_group_instruction) as group_instruction latest(status) as group_status by group_id]

  1. If any NEAP action rules has the following condition Create SNOW ticket when number of events is equal to 1, there is a chance that the issue will occasionally come up due to a race condition between the initialization of the episode and the creation of the SNOW ticket.
    1. Change the activation criteria to when number of events is equal to 2 (or anything else)
    2. Increase action_execution_delay in itsi_rules_engine.properties to a higher value (i.e. 100)
  2. Restart Splunk.

2021-12-07 ITSI-20343 Impacted Services and KPIs do not appear in Episode Review when using Teams functionality

Workaround:
Create/edit Template:SA-ITOA/local/macros.conf and add the following two stanzas:

{noformat}# Take in a string of team_keys in the format of '(sec_grp="itsi_team_key") OR (sec_grp="itsi_team_key")' and returns a filter of service_ids [itsi_events_compare_teams(1)] args = itsi_team_id_list definition = search (service_ids=*null*) OR (NOT service_ids=*) OR [|inputlookup itsi_services_in_team_lookup where ($itsi_team_id_list$) | rename _key as service_ids | eval service_ids="*".service_ids."*" | fields service_ids]

  1. Take in a string of team_keys in the format of '(sec_grp="itsi_team_key") OR (sec_grp="itsi_team_key")' and returns a filter of itsi_service_ids

[itsi_groups_compare_teams(1)] args = itsi_team_id_list definition = search (itsi_service_ids=*null*) OR (NOT itsi_service_ids=*) OR [|inputlookup itsi_services_in_team_lookup where ($itsi_team_id_list$) | rename _key as itsi_service_ids | eval itsi_service_ids="*".itsi_service_ids."*" | fields itsi_service_ids]{noformat}

2021-12-03 ITSI-20314 Episode not being marked as inactive when bulk close is used
2021-10-20 ITSI-19415 On Windows server, more than 1 rules engines processes are spawned at a time.

Workaround:
The root cause is the splunk phased_execution_mode. Edit the limits.conf file and add the line:  

[search]
phased_execution_mode = auto
2021-01-21 ITSI-13167 On Safari, there is a 10 to 15 second delay when editing a Notable Event Aggregation Policy using the ServiceNow action

Notable Event Aggregation Policies

Date filed Issue number Description
2022-12-20 ITSI-27751 Episode Review arbitrary search filter with AND & OR conditions fail to match events under certain scenarios

Workaround:
Avoid using brackets (), extra whitespaces, the operator !=, and double quotes "" in the search filter
2022-01-25 ITSI-21269 The grouping of Bidirectional Ticketing events sets the episode KV store state to faulty values
2022-01-05 ITSI-20978 ServiceNow bidirectional integration resets the episode title, status, severity, and owner.

Workaround:
#In etc/app/itsi/local/savedsearches.conf, replace the Bidirectional Ticketing clause with the following:

\[Bidirectional Ticketing] action.itsi_event_generator.param.description = %group_description% action.itsi_event_generator.param.itsi_instruction = %group_instruction% action.itsi_event_generator.param.owner = %group_assignee% action.itsi_event_generator.param.severity = %group_severity% action.itsi_event_generator.param.status = %group_status% action.itsi_event_generator.param.title = %group_title% disabled = 0 dispatch.earliest_time = -4h search = | `itsi_bidirectional_ticketing(main,90,itsi_tracked_alerts)`\\ | join group_id\\     \[ search `itsi_event_management_group_index_with_state("")` | rename itsi_group_id as group_id | stats latest(itsi_group_title) as group_title latest(severity) as group_severity latest(owner) as group_assignee latest(itsi_group_description) as group_description latest(itsi_group_instruction) as group_instruction latest(status) as group_status by group_id]

  1. If any NEAP action rules has the following condition Create SNOW ticket when number of events is equal to 1, there is a chance that the issue will occasionally come up due to a race condition between the initialization of the episode and the creation of the SNOW ticket.
    1. Change the activation criteria to when number of events is equal to 2 (or anything else)
    2. Increase action_execution_delay in itsi_rules_engine.properties to a higher value (i.e. 100)
  2. Restart Splunk.

2021-12-07 ITSI-20343 Impacted Services and KPIs do not appear in Episode Review when using Teams functionality

Workaround:
Create/edit Template:SA-ITOA/local/macros.conf and add the following two stanzas:

{noformat}# Take in a string of team_keys in the format of '(sec_grp="itsi_team_key") OR (sec_grp="itsi_team_key")' and returns a filter of service_ids [itsi_events_compare_teams(1)] args = itsi_team_id_list definition = search (service_ids=*null*) OR (NOT service_ids=*) OR [|inputlookup itsi_services_in_team_lookup where ($itsi_team_id_list$) | rename _key as service_ids | eval service_ids="*".service_ids."*" | fields service_ids]

  1. Take in a string of team_keys in the format of '(sec_grp="itsi_team_key") OR (sec_grp="itsi_team_key")' and returns a filter of itsi_service_ids

[itsi_groups_compare_teams(1)] args = itsi_team_id_list definition = search (itsi_service_ids=*null*) OR (NOT itsi_service_ids=*) OR [|inputlookup itsi_services_in_team_lookup where ($itsi_team_id_list$) | rename _key as itsi_service_ids | eval itsi_service_ids="*".itsi_service_ids."*" | fields itsi_service_ids]{noformat}

2021-12-03 ITSI-20314 Episode not being marked as inactive when bulk close is used
2021-10-20 ITSI-19415 On Windows server, more than 1 rules engines processes are spawned at a time.

Workaround:
The root cause is the splunk phased_execution_mode. Edit the limits.conf file and add the line:  

[search]
phased_execution_mode = auto
2021-01-21 ITSI-13167 On Safari, there is a 10 to 15 second delay when editing a Notable Event Aggregation Policy using the ServiceNow action

Glass Table

Date filed Issue number Description
2021-12-15 ITSI-20703 Glass Tables only load the first 100 fetched services and their respective KPIs and after loading those objects, no subsequent objects are populated with data, having "Service Unavailable" appearing instead.

KPI Base Searches

Date filed Issue number Description
2022-10-05 ITSI-26497 app/itsi/kpi_base_searches_lister error

Workaround:
N/A

KPI Search Calculation

Date filed Issue number Description
2022-05-31 ITSI-24437 KPI with split by entity stops working after upgrade to 4.11.5.

Workaround:
This command seems to get the KPI calculation going again:

/opt/splunk/bin/splunk cmd python /opt/splunk/etc/apps/SA-ITOA/bin/kvstore_to_json.py -m 4

2022-04-28 ITSI-23284 Deleted KPI lanes still showing in deep dive when the URL is refreshed.
2022-01-10 ITSI-21013 With custom indexes, when creating new KPI, the backfill checks look to the default itsi_summary instead of the custom one, causing potentially extra backfill.

Role Based Access Controls

Date filed Issue number Description
2022-12-19 ITSI-27734 Different users with same role itoa_team_admin cannot modify saved service analyzer.

Workaround:
Upgrade ITSI to version 4.13.x or 4.15.0
2021-12-14 ITSI-20605, ITSI-22366 Occasionally after ITSI upgrade, non-admin users get Oops Page - local.meta corrupted during the upgrade

Workaround:
Clean up all permissions on ITSI views in itsi/metadata/local.meta (and sync on SHC)

The workaround is to clean up the stanza in local.meta on the all the SH. remove all the stanza like \[views/....]  that have no valid access settings, (access = delete : \[  ], read : \[  ], write : \[  ]) and that are not custom views from your users.

As they may be many, to confirm, you can compare to the list in default.meta And you also can look at the modtime field in the stanza, as they are probably all identical.

Service Analyzer

Date filed Issue number Description
2023-02-17 ITSI-28826 Changes to health score color values in threshold_labels.conf do not appear in the service analyzer.
2022-12-19 ITSI-27734 Different users with same role itoa_team_admin cannot modify saved service analyzer.

Workaround:
Upgrade ITSI to version 4.13.x or 4.15.0
2022-02-17 ITSI-22146 Different users with same role itoa_team_admin cannot modify saved service analyzer.

Workaround:
Upgrade ITSI to version 4.13.x or 4.15.0

Uncategorized issues

Date filed Issue number Description
2023-01-09 ITSI-27961 Bidirectional Ticketing Correlation Search hits "subsearch limit of 50000 reached" when the collection itsi_notable_event_ticketing has more than 50000 entries

Workaround:
# Navigate to ITSI -> Configuration -> Correlation Searches
  1. Click on Bidirectional Ticketing
  2. Paste the following search in the Search field and then click on Save. Also enable the CS if it has been disabled

{noformat}| datamodel Ticket_Management Incident search | rename All_Ticket_Management.ticket_id as ticket_id | join ticket_id [search sourcetype="snow:incident" index="<snow_index>" | where _indextime > now() - <max_lookback_time>] | lookup itsi_notable_event_external_ticket tickets.ticket_id as ticket_id OUTPUTNEW tickets.ticket_system event_id | where isnotnull(event_id) | rename tickets.* as * | eventstats values(event_id) as group_id last(ticket_system) as ticket_system by ticket_id | fields - dv_* | table * | makemv group_id | mvexpand group_id | eval bidirectional_ticketing=1, snow_hash = number + "!" + group_id + "!" + sys_updated_on | search NOT [| search index="itsi_tracked_alerts" | fields snow_hash] | dedup snow_hash{noformat}

Change the placeholders {{<snow_index>}} and {{<max_lookback_time>}} in the above search with values according to the customer's requirements

2022-03-24 ITSI-22641 Premium features disabled because the ITSI license checker is not finding all the valid licenses, when they are more than 30 licenses installed

Workaround:
If the customer has more than 30 licenses, remove the expired ones to keep the list short.
2022-03-03 ITSI-22366, ITSI-20605 Occasionally after upgrade to ITSI 4.9.*, non-admin users get Oops Page - local.meta corrupted during the upgrade - Dev Fix Work

Workaround:
Clean up all permissions on ITSI views in itsi/metadata/local.meta (and sync on SHC)

The workaround is to clean up the stanza in local.meta on the all the SH. remove all the stanza like \[views/....]  that have no valid access settings, (access = delete : \[  ], read : \[  ], write : \[  ]) and that are not custom views from your users.

As they may be many, to confirm, you can compare to the list in default.meta And you also can look at the modtime field in the stanza, as they are probably all identical.

2022-01-31 ITSI-21357 Critical issue if the Splunk Add-On for Windows and Windows forwarder (from ITSI Data Integration) are installed on same machine

Workaround:
To resolve the conflict, add disabled = 0 for all seven stanzas in input.conf file for the universal forwarder. You can find the universal forwarder input.conf file in this location: C:\\Program Files\\SplunkUniversalForwarder\\etc\\apps\\SplunkUniversalForwarder\\local\\input.conf.
  • \[perfmon://CPU]
  • \[perfmon://LogicalDisk]
  • \[perfmon://Memory]
  • \[perfmon://Network]
  • \[perfmon://PhysicalDisk]
  • \[perfmon://Process]
  • \[perfmon://System]
2022-01-25 ITSI-21264 ITSI dev license - second dummy license does not work
2021-12-23 ITSI-20846 Bidirectional ticketing events being picked up as notable events and retitled as Ticket Event in episodes

Workaround:
# In etc/app/itsi/local/savedsearches.conf, replace the Bidirectional Ticketing clause with the following:

{noformat}[Bidirectional Ticketing] action.itsi_event_generator.param.description = %group_description% action.itsi_event_generator.param.itsi_instruction = %group_instruction% action.itsi_event_generator.param.owner = %group_assignee% action.itsi_event_generator.param.severity = %group_severity% action.itsi_event_generator.param.status = %group_status% action.itsi_event_generator.param.title = %group_title% disabled = 0 dispatch.earliest_time = -4h search = | `itsi_bidirectional_ticketing(main,90,itsi_tracked_alerts)`\ | join group_id\ 

   [ search `itsi_event_management_group_index_with_state("")` | rename itsi_group_id as group_id | stats latest(itsi_group_title) as group_title latest(itsi_group_severity) as group_severity latest(itsi_group_assignee) as group_assignee latest(itsi_group_description) as group_description latest(itsi_group_instruction) as group_instruction by group_id]

{noformat}

  1. Restart Splunk

This should help mitigate the issue while the code fix is in progress. If any of their NEAP action rules has the following condition Template:Create SNOW ticket when number of events is equal to 1, there is a chance that the issue will occasionally come up due to a race condition between the initialization of the episode and the creation of the SNOW ticket.

In they notice this occurring, they can either:

2021-10-25 ITSI-19489 The Next Scheduled Time for entity management policies is based on the system time zone, instead of the user's current time zone.
2021-09-09 ITSI-18800 When you add ITSI instances as search peers to another Splunk instance, the peers might be disabled after 72 hours. This is because the ITSI licenses are flagged as duplicates on the search peers.

Workaround:
#Go to the node search peer manager node.
  1. Identify the Splunk licenses (Enterprise, ITSI, non-ITSI) currently installed. Ignore licenses under the *IT Service Intelligence Internals DO NOT COPY* stack.
  1. Navigate to http://LM_IP/en-US/manager/system/licensing/licenses and check if the AllowDuplicateKeys capability is enabled for each of the license identified in step 1.
  1. If not enabled, procure a new license from Splunk support and replace it.
  1. Make sure all licenses in the stack have the capability enabled.
  1. Restart Splunk.
2021-09-01 ITSI-18709 ITSI redirects to suite_redirect 500 Internal Server Error - because of python library isolation between apps

Workaround:
Step 1: Identify all the splunklib directories within the splunk apps directory using command find . -name 'splunklib' | xargs -r ls -lah.

Step 2: For each directory listed in step 1, check if file six.py is present.

Step 3: Copy the six.py from an existing splunklib directory into all the missing directories.

Step 4: Clean the cached files using find . -name "*.pyc" -delete

Step 5: Restart Splunk on the ITE Work or ITSI search head.

2019-05-30 ITSI-3322 If you add a correlation search in ITSI which contains a sub-search returning into an eval, you get a message "Invalid search string: This search cannot be parsed when parse_only is set to true."

Workaround:
You can't use a sub-search returning into an eval in a correlation search. As a workaround, create and save a basic correlation search with all of the information you want outside of the search. Then as an admin user, go to Settings > Searches, reports, and alerts and open the correlation search you just created. Add the sub-search you were trying to add there.
Last modified on 11 January, 2024
PREVIOUS
Fixed issues in Splunk IT Service Intelligence 		  NEXT
Removed features in Splunk IT Service Intelligence

This documentation applies to the following versions of Splunk® IT Service Intelligence: 4.11.2

Was this documentation topic helpful?


You must be logged into splunk.com in order to post comments. Log in now.

Please try to keep this discussion focused on the content covered in this documentation topic. If you have a more general question about Splunk functionality or are experiencing a difficulty with Splunk, consider posting a question to Splunkbase Answers.

0 out of 1000 Characters