Splunk® IT Service Intelligence

Release Notes

Acrobat logo Download manual as PDF


Splunk IT Service Intelligence (ITSI) version 4.11.x reached its End of Life on December 6, 2023. See the Splunk Software Support Policy for details. For information about upgrading to a supported version, see Before you upgrade IT Service Intelligence.
This documentation does not apply to the most recent version of Splunk® IT Service Intelligence. For documentation on the most recent version, go to the latest release.
Acrobat logo Download topic as PDF

Known issues in Splunk IT Service Intelligence

IT Service Intelligence (ITSI) version 4.11.0 has the following known issues and workarounds.

Adaptive Thresholding

Date filed Issue number Description
2021-11-05 ITSI-19663 Updating a KPI threshold policy within a service template causes the thresholds of all existing KPIs that use adaptive thresholds to get reset

Workaround:
Temporary workaround to avoid false alerts: 
# Put services that are linked to the service template into maintenance mode
# Make KPI threshold changes within the service template and push out
# Wait to make sure all services are synced
# Manually run the itsi_at_search_kpi_minusXd to recreate the adaptive threshold models
# Disable maintenance mode for false alerts

Backup/Restore and Migration Issues

Date filed Issue number Description
2021-12-06 ITSI-20325 When a backup .ZIP file includes a base search with a title that is over 100 characters, the backup restore job will fail.
2021-12-02 ITSI-20308 Errors found in the migration log while upgrading to 4.11.0
2021-10-13 ITSI-19215 Customer is getting a lot of errors related to "Could not find object id=itsi_entity_dashboard_drilldown" after installation of IT Essentials Work

Workaround:
Upgrade to ITE-Work version 4.12 and later

Notable Events

Date filed Issue number Description
2023-06-29 ITSI-31192 All Events tab does not render default columns if they are not present in NEAP JSON definition

Workaround:
# Use the latest ITSI Backup file to edit the NEAP JSON definition and remove the property Template:All events columns and restore the backup.
  1. Go to Episode Review page and add back all the desired columns
2023-02-08 ITSI-28707 Color for custom severity is not displayed correctly in Correlation Search Builder, Notable Event Aggregation Policy Editor and Episode Review page
2023-01-16 ITSI-28046 Alert action configuration UI not loaded in ITSI when the count of alert actions exceed 30

Workaround:
Keep the count of alert actions in the instance below 30
2023-01-12 ITSI-28015 The episode link in "Share Episode" does not get updated in right click menu
2022-12-20 ITSI-27751 Episode Review arbitrary search filter with AND & OR conditions fail to match events under certain scenarios

Workaround:
Avoid using brackets (), extra whitespaces, the operator !=, and double quotes "" in the search filter
2022-12-11 ITSI-27640 Event Analytics Monitoring dashboard does not list all NEAP

Workaround:
in Event Analytics Monitoring Dashboard → Aggregation Policy panel → Edit → edit the Aggregation policy search under Dynamic Options:

{noformat}| rest servicesNS/nobody/SA-ITOA/event_management_interface/notable_event_aggregation_policy splunk_server=* report_as=text 

 | spath input=value path={}.title | spath input=value path={}._key | rename {}.title as title 
 | rename {}._key as key | eval zipped=mvzip('key', 'title') | mvexpand zipped | eval zipped=split(zipped, ",") 
 | eval itsi_policy_id=mvindex(zipped,0), policy_title=mvindex(zipped, 1)

{noformat}

2022-07-06 ITSI-24871 NEAP breaking criteria not obeying OR condition when time based conditions are selected

Workaround:
Keep the time based breaking conditions i.e. "_If this episode existed for:_ " and "_if the flow of events in the episode paused for:_" at the end of the OR conditions and the "_The following event occurs_" condition as first.
2022-01-25 ITSI-21269 The grouping of Bidirectional Ticketing events sets the episode KV store state to faulty values
2022-01-05 ITSI-20978 ServiceNow bidirectional integration resets the episode title, status, severity, and owner.

Workaround:
#In etc/app/itsi/local/savedsearches.conf, replace the Bidirectional Ticketing clause with the following:

\[Bidirectional Ticketing] action.itsi_event_generator.param.description = %group_description% action.itsi_event_generator.param.itsi_instruction = %group_instruction% action.itsi_event_generator.param.owner = %group_assignee% action.itsi_event_generator.param.severity = %group_severity% action.itsi_event_generator.param.status = %group_status% action.itsi_event_generator.param.title = %group_title% disabled = 0 dispatch.earliest_time = -4h search = | `itsi_bidirectional_ticketing(main,90,itsi_tracked_alerts)`\\ | join group_id\\     \[ search `itsi_event_management_group_index_with_state("")` | rename itsi_group_id as group_id | stats latest(itsi_group_title) as group_title latest(severity) as group_severity latest(owner) as group_assignee latest(itsi_group_description) as group_description latest(itsi_group_instruction) as group_instruction latest(status) as group_status by group_id]

  1. If any NEAP action rules has the following condition Create SNOW ticket when number of events is equal to 1, there is a chance that the issue will occasionally come up due to a race condition between the initialization of the episode and the creation of the SNOW ticket.
    1. Change the activation criteria to when number of events is equal to 2 (or anything else)
    2. Increase action_execution_delay in itsi_rules_engine.properties to a higher value (i.e. 100)
  2. Restart Splunk.

2021-12-10 ITSI-20467 Can only see the first 20 NEAPs on NEAP lister

Workaround:
Use the filter to narrow down to 20 NEAPs or less
2021-12-07 ITSI-20343 Impacted Services and KPIs do not appear in Episode Review when using Teams functionality

Workaround:
Create/edit Template:SA-ITOA/local/macros.conf and add the following two stanzas:

{noformat}# Take in a string of team_keys in the format of '(sec_grp="itsi_team_key") OR (sec_grp="itsi_team_key")' and returns a filter of service_ids [itsi_events_compare_teams(1)] args = itsi_team_id_list definition = search (service_ids=*null*) OR (NOT service_ids=*) OR [|inputlookup itsi_services_in_team_lookup where ($itsi_team_id_list$) | rename _key as service_ids | eval service_ids="*".service_ids."*" | fields service_ids]

  1. Take in a string of team_keys in the format of '(sec_grp="itsi_team_key") OR (sec_grp="itsi_team_key")' and returns a filter of itsi_service_ids

[itsi_groups_compare_teams(1)] args = itsi_team_id_list definition = search (itsi_service_ids=*null*) OR (NOT itsi_service_ids=*) OR [|inputlookup itsi_services_in_team_lookup where ($itsi_team_id_list$) | rename _key as itsi_service_ids | eval itsi_service_ids="*".itsi_service_ids."*" | fields itsi_service_ids]{noformat}

2021-12-03 ITSI-20314 Episode not being marked as inactive when bulk close is used
2021-10-20 ITSI-19415 On Windows server, more than 1 rules engines processes are spawned at a time.

Workaround:
The root cause is the splunk phased_execution_mode. Edit the limits.conf file and add the line:  

[search]
phased_execution_mode = auto
2021-01-21 ITSI-13167 On Safari, there is a 10 to 15 second delay when editing a Notable Event Aggregation Policy using the ServiceNow action

Notable Event Aggregation Policies

Date filed Issue number Description
2023-06-29 ITSI-31192 All Events tab does not render default columns if they are not present in NEAP JSON definition

Workaround:
# Use the latest ITSI Backup file to edit the NEAP JSON definition and remove the property Template:All events columns and restore the backup.
  1. Go to Episode Review page and add back all the desired columns
2023-02-08 ITSI-28707 Color for custom severity is not displayed correctly in Correlation Search Builder, Notable Event Aggregation Policy Editor and Episode Review page
2023-01-16 ITSI-28046 Alert action configuration UI not loaded in ITSI when the count of alert actions exceed 30

Workaround:
Keep the count of alert actions in the instance below 30
2023-01-12 ITSI-28015 The episode link in "Share Episode" does not get updated in right click menu
2022-12-20 ITSI-27751 Episode Review arbitrary search filter with AND & OR conditions fail to match events under certain scenarios

Workaround:
Avoid using brackets (), extra whitespaces, the operator !=, and double quotes "" in the search filter
2022-12-11 ITSI-27640 Event Analytics Monitoring dashboard does not list all NEAP

Workaround:
in Event Analytics Monitoring Dashboard → Aggregation Policy panel → Edit → edit the Aggregation policy search under Dynamic Options:

{noformat}| rest servicesNS/nobody/SA-ITOA/event_management_interface/notable_event_aggregation_policy splunk_server=* report_as=text 

 | spath input=value path={}.title | spath input=value path={}._key | rename {}.title as title 
 | rename {}._key as key | eval zipped=mvzip('key', 'title') | mvexpand zipped | eval zipped=split(zipped, ",") 
 | eval itsi_policy_id=mvindex(zipped,0), policy_title=mvindex(zipped, 1)

{noformat}

2022-07-06 ITSI-24871 NEAP breaking criteria not obeying OR condition when time based conditions are selected

Workaround:
Keep the time based breaking conditions i.e. "_If this episode existed for:_ " and "_if the flow of events in the episode paused for:_" at the end of the OR conditions and the "_The following event occurs_" condition as first.
2022-01-25 ITSI-21269 The grouping of Bidirectional Ticketing events sets the episode KV store state to faulty values
2022-01-05 ITSI-20978 ServiceNow bidirectional integration resets the episode title, status, severity, and owner.

Workaround:
#In etc/app/itsi/local/savedsearches.conf, replace the Bidirectional Ticketing clause with the following:

\[Bidirectional Ticketing] action.itsi_event_generator.param.description = %group_description% action.itsi_event_generator.param.itsi_instruction = %group_instruction% action.itsi_event_generator.param.owner = %group_assignee% action.itsi_event_generator.param.severity = %group_severity% action.itsi_event_generator.param.status = %group_status% action.itsi_event_generator.param.title = %group_title% disabled = 0 dispatch.earliest_time = -4h search = | `itsi_bidirectional_ticketing(main,90,itsi_tracked_alerts)`\\ | join group_id\\     \[ search `itsi_event_management_group_index_with_state("")` | rename itsi_group_id as group_id | stats latest(itsi_group_title) as group_title latest(severity) as group_severity latest(owner) as group_assignee latest(itsi_group_description) as group_description latest(itsi_group_instruction) as group_instruction latest(status) as group_status by group_id]

  1. If any NEAP action rules has the following condition Create SNOW ticket when number of events is equal to 1, there is a chance that the issue will occasionally come up due to a race condition between the initialization of the episode and the creation of the SNOW ticket.
    1. Change the activation criteria to when number of events is equal to 2 (or anything else)
    2. Increase action_execution_delay in itsi_rules_engine.properties to a higher value (i.e. 100)
  2. Restart Splunk.

2021-12-10 ITSI-20467 Can only see the first 20 NEAPs on NEAP lister

Workaround:
Use the filter to narrow down to 20 NEAPs or less
2021-12-07 ITSI-20343 Impacted Services and KPIs do not appear in Episode Review when using Teams functionality

Workaround:
Create/edit Template:SA-ITOA/local/macros.conf and add the following two stanzas:

{noformat}# Take in a string of team_keys in the format of '(sec_grp="itsi_team_key") OR (sec_grp="itsi_team_key")' and returns a filter of service_ids [itsi_events_compare_teams(1)] args = itsi_team_id_list definition = search (service_ids=*null*) OR (NOT service_ids=*) OR [|inputlookup itsi_services_in_team_lookup where ($itsi_team_id_list$) | rename _key as service_ids | eval service_ids="*".service_ids."*" | fields service_ids]

  1. Take in a string of team_keys in the format of '(sec_grp="itsi_team_key") OR (sec_grp="itsi_team_key")' and returns a filter of itsi_service_ids

[itsi_groups_compare_teams(1)] args = itsi_team_id_list definition = search (itsi_service_ids=*null*) OR (NOT itsi_service_ids=*) OR [|inputlookup itsi_services_in_team_lookup where ($itsi_team_id_list$) | rename _key as itsi_service_ids | eval itsi_service_ids="*".itsi_service_ids."*" | fields itsi_service_ids]{noformat}

2021-12-03 ITSI-20314 Episode not being marked as inactive when bulk close is used
2021-10-20 ITSI-19415 On Windows server, more than 1 rules engines processes are spawned at a time.

Workaround:
The root cause is the splunk phased_execution_mode. Edit the limits.conf file and add the line:  

[search]
phased_execution_mode = auto
2021-01-21 ITSI-13167 On Safari, there is a 10 to 15 second delay when editing a Notable Event Aggregation Policy using the ServiceNow action

Glass Table

Date filed Issue number Description
2021-12-17 ITSI-20748 Service Swapping weirdness on Glass Table
2021-12-15 ITSI-20703 Glass Tables only load the first 100 fetched services and their respective KPIs and after loading those objects, no subsequent objects are populated with data, having "Service Unavailable" appearing instead.

KPI Base Searches

Date filed Issue number Description
2022-10-05 ITSI-26497 app/itsi/kpi_base_searches_lister error

Workaround:
N/A

KPI Search Calculation

Date filed Issue number Description
2023-02-24 ITSI-28886 mod_time and retirable appear as a metric_name in itsi_summary_metrics and unnecessarily creates extra datapoints
2022-05-31 ITSI-24437 KPI with split by entity stops working after upgrade to 4.11.5.

Workaround:
This command seems to get the KPI calculation going again:

/opt/splunk/bin/splunk cmd python /opt/splunk/etc/apps/SA-ITOA/bin/kvstore_to_json.py -m 4

2022-04-28 ITSI-23284 Deleted KPI lanes still showing in deep dive when the URL is refreshed.
2022-01-10 ITSI-21013 With custom indexes, when creating new KPI, the backfill checks look to the default itsi_summary instead of the custom one, causing potentially extra backfill.

Service Definition

Date filed Issue number Description
2021-10-07 ITSI-19172 With large numbers of entities on a system, Service Definition dimension-value multiselect freezes

Service Health Score

Date filed Issue number Description
2022-09-28 ITSI-26376 Large number of KPI caused the service_health_metrics_monitor sub search to hit the 50000 default limit, causing discrepancies in values in Service Health Score alert_level in itsi_summary_metrics versus itsi_summary indexes.

Workaround:
Increase the limits.conf to adjust to the total number of KPIs in the subsearch of service_health_metrics_monitor. See example for a customer with 50000-70000 KPI objects.

{{[join] }} Template:Subsearch maxout = 75000 {{#default was 50000 }} {{[searchresults] }} Template:Maxresultrows = 75000 {{ # default was 50000}}

Uncategorized issues

Date filed Issue number Description
2023-01-09 ITSI-27961 Bidirectional Ticketing Correlation Search hits "subsearch limit of 50000 reached" when the collection itsi_notable_event_ticketing has more than 50000 entries

Workaround:
# Navigate to ITSI -> Configuration -> Correlation Searches
  1. Click on Bidirectional Ticketing
  2. Paste the following search in the Search field and then click on Save. Also enable the CS if it has been disabled

{noformat}| datamodel Ticket_Management Incident search | rename All_Ticket_Management.ticket_id as ticket_id | join ticket_id [search sourcetype="snow:incident" index="<snow_index>" | where _indextime > now() - <max_lookback_time>] | lookup itsi_notable_event_external_ticket tickets.ticket_id as ticket_id OUTPUTNEW tickets.ticket_system event_id | where isnotnull(event_id) | rename tickets.* as * | eventstats values(event_id) as group_id last(ticket_system) as ticket_system by ticket_id | fields - dv_* | table * | makemv group_id | mvexpand group_id | eval bidirectional_ticketing=1, snow_hash = number + "!" + group_id + "!" + sys_updated_on | search NOT [| search index="itsi_tracked_alerts" | fields snow_hash] | dedup snow_hash{noformat}

Change the placeholders {{<snow_index>}} and {{<max_lookback_time>}} in the above search with values according to the customer's requirements

2022-09-06 ITSI-26046 NumberFormatException causing Episodes to remain unbroken when NEAP is time-based and Episode Severity set to Same as Highest Severity

Workaround:
The customer will be able to manually close the episodes.

IMPORTANT: the outputlookup command is dangerous when used with the kvstore. It will overwrite the contents of the entire kvstore collection with the search results if the Template:Append=true flag is not set. The customer should make a backup before running the command.

Search to generate the objects to push to kvstore. Please run this search for the past 30 days.

{noformat}`itsi_event_management_group_index` | stats latest(owner) as owner, latest(severity) as severity, latest(status) as status, latest(itsi_instruction) as instruction by itsi_group_id | eval index_owner=owner, index_severity=severity, index_status=status, event_identifier_hash=itsi_group_id | fields index_owner, index_severity, index_status, itsi_group_id, instruction, event_identifier_hash | eval _key=itsi_group_id | lookup itsi_notable_group_system_lookup _key OUTPUT mod_time | lookup itsi_notable_group_user_lookup _key OUTPUT owner severity status | search NOT status=* AND mod_time=* | eval owner=index_owner, severity=index_severity, status=index_status, object_type="notable_group_user" | fields - index_owner, index_severity, index_status

{noformat}

If results look correct append the following Template:Outputlookup command and re-run search:

{noformat}| outputlookup itsi_notable_group_user_lookup append=true key_field=itsi_group_id{noformat}

This search should ideally update these Episodes:

"2a617192-1858-4219-aba8-ed7b777f3035" "ad3ec87e-05c2-4b1c-8ca9-c854ac6f6725" "ccfa9689-a4e8-460e-a001-45e6891361a8"

2022-03-24 ITSI-22641 Premium features disabled because the ITSI license checker is not finding all the valid licenses, when they are more than 30 licenses installed

Workaround:
If the customer has more than 30 licenses, remove the expired ones to keep the list short.
2022-02-11 ITSI-22056 Incorrect URL encoding when navigating in Entity Overview
2022-01-31 ITSI-21357 Critical issue if the Splunk Add-On for Windows and Windows forwarder (from ITSI Data Integration) are installed on same machine

Workaround:
To resolve the conflict, add disabled = 0 for all seven stanzas in input.conf file for the universal forwarder. You can find the universal forwarder input.conf file in this location: C:\\Program Files\\SplunkUniversalForwarder\\etc\\apps\\SplunkUniversalForwarder\\local\\input.conf.
  • \[perfmon://CPU]
  • \[perfmon://LogicalDisk]
  • \[perfmon://Memory]
  • \[perfmon://Network]
  • \[perfmon://PhysicalDisk]
  • \[perfmon://Process]
  • \[perfmon://System]
2021-12-23 ITSI-20846 Bidirectional ticketing events being picked up as notable events and retitled as Ticket Event in episodes

Workaround:
# In etc/app/itsi/local/savedsearches.conf, replace the Bidirectional Ticketing clause with the following:

{noformat}[Bidirectional Ticketing] action.itsi_event_generator.param.description = %group_description% action.itsi_event_generator.param.itsi_instruction = %group_instruction% action.itsi_event_generator.param.owner = %group_assignee% action.itsi_event_generator.param.severity = %group_severity% action.itsi_event_generator.param.status = %group_status% action.itsi_event_generator.param.title = %group_title% disabled = 0 dispatch.earliest_time = -4h search = | `itsi_bidirectional_ticketing(main,90,itsi_tracked_alerts)`\ | join group_id\ 

   [ search `itsi_event_management_group_index_with_state("")` | rename itsi_group_id as group_id | stats latest(itsi_group_title) as group_title latest(itsi_group_severity) as group_severity latest(itsi_group_assignee) as group_assignee latest(itsi_group_description) as group_description latest(itsi_group_instruction) as group_instruction by group_id]

{noformat}

  1. Restart Splunk

This should help mitigate the issue while the code fix is in progress. If any of their NEAP action rules has the following condition Template:Create SNOW ticket when number of events is equal to 1, there is a chance that the issue will occasionally come up due to a race condition between the initialization of the episode and the creation of the SNOW ticket.

In they notice this occurring, they can either:

2021-12-14 ITSI-20653 Linux Data Integrations are non-functional

Workaround:
If the user doesn't need logs, they can run the installation script with logs deselected, and the UF will not be installed. Otherwise, the user can substitute a link to the tar-file version of the UF that fits from [1]. The specific link depends on which version and which distro they want to use.
2021-10-25 ITSI-19489 The Next Scheduled Time for entity management policies is based on the system time zone, instead of the user's current time zone.
2021-09-09 ITSI-18800 When you add ITSI instances as search peers to another Splunk instance, the peers might be disabled after 72 hours. This is because the ITSI licenses are flagged as duplicates on the search peers.

Workaround:
#Go to the node search peer manager node.
  1. Identify the Splunk licenses (Enterprise, ITSI, non-ITSI) currently installed. Ignore licenses under the *IT Service Intelligence Internals DO NOT COPY* stack.
  1. Navigate to http://LM_IP/en-US/manager/system/licensing/licenses and check if the AllowDuplicateKeys capability is enabled for each of the license identified in step 1.
  1. If not enabled, procure a new license from Splunk support and replace it.
  1. Make sure all licenses in the stack have the capability enabled.
  1. Restart Splunk.
2021-09-01 ITSI-18709 ITSI redirects to suite_redirect 500 Internal Server Error - because of python library isolation between apps

Workaround:
Step 1: Identify all the splunklib directories within the splunk apps directory using command find . -name 'splunklib' | xargs -r ls -lah.

Step 2: For each directory listed in step 1, check if file six.py is present.

Step 3: Copy the six.py from an existing splunklib directory into all the missing directories.

Step 4: Clean the cached files using find . -name "*.pyc" -delete

Step 5: Restart Splunk on the ITE Work or ITSI search head.

Last modified on 12 January, 2024
PREVIOUS
Fixed issues in Splunk IT Service Intelligence 		  NEXT
Removed features in Splunk IT Service Intelligence

This documentation applies to the following versions of Splunk® IT Service Intelligence: 4.11.0

Was this documentation topic helpful?


You must be logged into splunk.com in order to post comments. Log in now.

Please try to keep this discussion focused on the content covered in this documentation topic. If you have a more general question about Splunk functionality or are experiencing a difficulty with Splunk, consider posting a question to Splunkbase Answers.

0 out of 1000 Characters