Splunk® IT Service Intelligence

Release Notes

Acrobat logo Download manual as PDF


This documentation does not apply to the most recent version of ITSI. Click here for the latest version.
Acrobat logo Download topic as PDF

Known issues in Splunk IT Service Intelligence

IT Service Intelligence (ITSI) version 4.11.3 has the following known issues and workarounds.

Adaptive Thresholding

Date filed Issue number Description
2022-02-10 ITSI-21921 Preview for aggregate thresholds takes a long time to load

Workaround:
From the Preferences modal, set the Time zone to the Default System Timezone.
2021-11-05 ITSI-19663 Updating a KPI threshold policy within a service template causes the thresholds of all existing KPIs that use adaptive thresholds to get reset

Workaround:
Temporary workaround to avoid false alerts: 
# Put services that are linked to the service template into maintenance mode
# Make KPI threshold changes within the service template and push out
# Wait to make sure all services are synced
# Manually run the itsi_at_search_kpi_minusXd to recreate the adaptive threshold models
# Disable maintenance mode for false alerts

Backup/Restore and Migration Issues

Date filed Issue number Description
2021-12-06 ITSI-20325 When a backup .ZIP file includes a base search with a title that is over 100 characters, the backup restore job will fail.
2021-12-02 ITSI-20308 Errors found in the migration log while upgrading to 4.11.0

Notable Events

Date filed Issue number Description
2022-01-25 ITSI-21269 The grouping of Bidirectional Ticketing events sets the episode KV store state to faulty values
2022-01-05 ITSI-20978 ServiceNow bidirectional integration resets the episode title, status, severity, and owner.

Workaround:
  1. In etc/app/itsi/local/savedsearches.conf, replace the Bidirectional Ticketing clause with the following: \[Bidirectional Ticketing] action.itsi_event_generator.param.description = %group_description% action.itsi_event_generator.param.itsi_instruction = %group_instruction% action.itsi_event_generator.param.owner = %group_assignee% action.itsi_event_generator.param.severity = %group_severity% action.itsi_event_generator.param.status = %group_status% action.itsi_event_generator.param.title = %group_title% disabled = 0 dispatch.earliest_time = -4h search = | `itsi_bidirectional_ticketing(main,90,itsi_tracked_alerts)`\\ | join group_id\\     \[ search `itsi_event_management_group_index_with_state("")` | rename itsi_group_id as group_id | stats latest(itsi_group_title) as group_title latest(severity) as group_severity latest(owner) as group_assignee latest(itsi_group_description) as group_description latest(itsi_group_instruction) as group_instruction latest(status) as group_status by group_id]
  2. If any NEAP action rules has the following condition Create SNOW ticket when number of events is equal to 1, there is a chance that the issue will occasionally come up due to a race condition between the initialization of the episode and the creation of the SNOW ticket.
    1. Change the activation criteria to when number of events is equal to 2 (or anything else)
    2. Increase action_execution_delay in itsi_rules_engine.properties to a higher value (i.e. 100)
  3. Restart Splunk.

2021-12-07 ITSI-20343 Impacted Services and KPIs do not appear in Episode Review when using Teams functionality

Workaround:
Create/edit Template:SA-ITOA/local/macros.conf and add the following two stanzas:

{noformat}# Take in a string of team_keys in the format of '(sec_grp="itsi_team_key") OR (sec_grp="itsi_team_key")' and returns a filter of service_ids [itsi_events_compare_teams(1)] args = itsi_team_id_list definition = search (service_ids=*null*) OR (NOT service_ids=*) OR [|inputlookup itsi_services_in_team_lookup where ($itsi_team_id_list$) | rename _key as service_ids | eval service_ids="*".service_ids."*" | fields service_ids]

  1. Take in a string of team_keys in the format of '(sec_grp="itsi_team_key") OR (sec_grp="itsi_team_key")' and returns a filter of itsi_service_ids

[itsi_groups_compare_teams(1)] args = itsi_team_id_list definition = search (itsi_service_ids=*null*) OR (NOT itsi_service_ids=*) OR [|inputlookup itsi_services_in_team_lookup where ($itsi_team_id_list$) | rename _key as itsi_service_ids | eval itsi_service_ids="*".itsi_service_ids."*" | fields itsi_service_ids]{noformat}

2021-12-03 ITSI-20314 Episode not being marked as inactive when bulk close is used
2021-10-20 ITSI-19415 On Windows server, more than 1 rules engines processes are spawned at a time.

Workaround:
The root cause is the splunk phased_execution_mode. Edit the limits.conf file and add the line:  

[search]
phased_execution_mode = auto
2021-01-21 ITSI-13167 On Safari, there is a 10 to 15 second delay when editing a Notable Event Aggregation Policy using the ServiceNow action

Notable Event Aggregation Policies

Date filed Issue number Description
2022-01-25 ITSI-21269 The grouping of Bidirectional Ticketing events sets the episode KV store state to faulty values
2022-01-05 ITSI-20978 ServiceNow bidirectional integration resets the episode title, status, severity, and owner.

Workaround:
  1. In etc/app/itsi/local/savedsearches.conf, replace the Bidirectional Ticketing clause with the following: \[Bidirectional Ticketing] action.itsi_event_generator.param.description = %group_description% action.itsi_event_generator.param.itsi_instruction = %group_instruction% action.itsi_event_generator.param.owner = %group_assignee% action.itsi_event_generator.param.severity = %group_severity% action.itsi_event_generator.param.status = %group_status% action.itsi_event_generator.param.title = %group_title% disabled = 0 dispatch.earliest_time = -4h search = | `itsi_bidirectional_ticketing(main,90,itsi_tracked_alerts)`\\ | join group_id\\     \[ search `itsi_event_management_group_index_with_state("")` | rename itsi_group_id as group_id | stats latest(itsi_group_title) as group_title latest(severity) as group_severity latest(owner) as group_assignee latest(itsi_group_description) as group_description latest(itsi_group_instruction) as group_instruction latest(status) as group_status by group_id]
  2. If any NEAP action rules has the following condition Create SNOW ticket when number of events is equal to 1, there is a chance that the issue will occasionally come up due to a race condition between the initialization of the episode and the creation of the SNOW ticket.
    1. Change the activation criteria to when number of events is equal to 2 (or anything else)
    2. Increase action_execution_delay in itsi_rules_engine.properties to a higher value (i.e. 100)
  3. Restart Splunk.

2021-12-07 ITSI-20343 Impacted Services and KPIs do not appear in Episode Review when using Teams functionality

Workaround:
Create/edit Template:SA-ITOA/local/macros.conf and add the following two stanzas:

{noformat}# Take in a string of team_keys in the format of '(sec_grp="itsi_team_key") OR (sec_grp="itsi_team_key")' and returns a filter of service_ids [itsi_events_compare_teams(1)] args = itsi_team_id_list definition = search (service_ids=*null*) OR (NOT service_ids=*) OR [|inputlookup itsi_services_in_team_lookup where ($itsi_team_id_list$) | rename _key as service_ids | eval service_ids="*".service_ids."*" | fields service_ids]

  1. Take in a string of team_keys in the format of '(sec_grp="itsi_team_key") OR (sec_grp="itsi_team_key")' and returns a filter of itsi_service_ids

[itsi_groups_compare_teams(1)] args = itsi_team_id_list definition = search (itsi_service_ids=*null*) OR (NOT itsi_service_ids=*) OR [|inputlookup itsi_services_in_team_lookup where ($itsi_team_id_list$) | rename _key as itsi_service_ids | eval itsi_service_ids="*".itsi_service_ids."*" | fields itsi_service_ids]{noformat}

2021-12-03 ITSI-20314 Episode not being marked as inactive when bulk close is used
2021-10-20 ITSI-19415 On Windows server, more than 1 rules engines processes are spawned at a time.

Workaround:
The root cause is the splunk phased_execution_mode. Edit the limits.conf file and add the line:  

[search]
phased_execution_mode = auto
2021-01-21 ITSI-13167 On Safari, there is a 10 to 15 second delay when editing a Notable Event Aggregation Policy using the ServiceNow action

Uncategorized issues

Date filed Issue number Description
2022-03-24 ITSI-22641 Premium features disabled because the ITSI license checker is not finding all the valid licenses, when they are more than 30 licenses installed

Workaround:
If the customer has more than 30 licenses, remove the expired ones to keep the list short.
2022-02-16 ITSI-22140 Sorted by in Episode Review does not work with "First Event Time"
2022-02-04 ITSI-21526 Endpoint for closing and breaking an episode does not show a warning when all the fields are not provided in payload
2022-01-31 ITSI-21357 Critical issue if the Splunk Add-On for Windows and Windows forwarder (from ITSI Data Integration) are installed on same machine

Workaround:
To resolve the conflict, add disabled = 0 for all seven stanzas in input.conf file for the universal forwarder. You can find the universal forwarder input.conf file in this location: C:\\Program Files\\SplunkUniversalForwarder\\etc\\apps\\SplunkUniversalForwarder\\local\\input.conf.
  • \[perfmon://CPU]
  • \[perfmon://LogicalDisk]
  • \[perfmon://Memory]
  • \[perfmon://Network]
  • \[perfmon://PhysicalDisk]
  • \[perfmon://Process]
  • \[perfmon://System]
2022-01-31 ITSI-21361 Bad regex in the ITSI Log Messages deduplicated panel of the ITSI health dashboard
2022-01-07 ITSI-21005 Discovery searches are not updating entities
2021-12-23 ITSI-20846 Bidirectional ticketing events being picked up as notable events and retitled as Ticket Event in episodes

Workaround:
# In etc/app/itsi/local/savedsearches.conf, replace the Bidirectional Ticketing clause with the following:

{noformat}[Bidirectional Ticketing] action.itsi_event_generator.param.description = %group_description% action.itsi_event_generator.param.itsi_instruction = %group_instruction% action.itsi_event_generator.param.owner = %group_assignee% action.itsi_event_generator.param.severity = %group_severity% action.itsi_event_generator.param.status = %group_status% action.itsi_event_generator.param.title = %group_title% disabled = 0 dispatch.earliest_time = -4h search = | `itsi_bidirectional_ticketing(main,90,itsi_tracked_alerts)`\ | join group_id\ 

   [ search `itsi_event_management_group_index_with_state("")` | rename itsi_group_id as group_id | stats latest(itsi_group_title) as group_title latest(itsi_group_severity) as group_severity latest(itsi_group_assignee) as group_assignee latest(itsi_group_description) as group_description latest(itsi_group_instruction) as group_instruction by group_id]

{noformat}

  1. Restart Splunk

This should help mitigate the issue while the code fix is in progress. If any of their NEAP action rules has the following condition Template:Create SNOW ticket when number of events is equal to 1, there is a chance that the issue will occasionally come up due to a race condition between the initialization of the episode and the creation of the SNOW ticket.

In they notice this occurring, they can either:

2021-12-14 ITSI-20605, ITSI-22366 Occasionally after upgrade to ITSI 4.9.*, non-admin users get Oops Page - local.meta corrupted during the upgrade

Workaround:
Clean up all permissions on ITSI views in itsi/metadata/local.meta (and sync on SHC)

The workaround is to clean up the stanza in local.meta on the all the SH. remove all the stanza like \[views/....]  that have no valid access settings, (access = delete : \[  ], read : \[  ], write : \[  ]) and that are not custom views from your users.

As they may be many, to confirm, you can compare to the list in default.meta And you also can look at the modtime field in the stanza, as they are probably all identical.

2021-10-25 ITSI-19489 The Next Scheduled Time for entity management policies is based on the system time zone, instead of the user's current time zone.
2021-09-09 ITSI-18800 When you add ITSI instances as search peers to another Splunk instance, the peers might be disabled after 72 hours. This is because the ITSI licenses are flagged as duplicates on the search peers.
2021-09-01 ITSI-18709 ITSI 4.9 redirects to suite_redirect 500 Internal Server Error - because of python library isolation between apps

Workaround:
#Identify all the splunklib directories within the splunk apps directory using command find . -name 'splunklib' | xargs -r ls -lah.
  1. For each directory listed in step 1, check if file six.py is present.
  1. Copy the six.py from an existing splunklib directory into all the missing directories.
  1. Clean the cached files using find . -name "*.pyc" -delete
  1. Reload the ITE Work app.

All ITSI Modules

Publication date Issue number Description
2017-03-21 ITOA-7585 When you bulk add services and an error caused by the racing condition occurs, the incorrect message "itsi_module does not exist" is displayed.
2017-03-07 MOD-979 KPIs do not have consistent backfill settings across all modules.
2017-01-17 MOD-452 The Analyze KPI button on the Service Details page is broken.
2017-01-17 MOD-402 The Export to PDF option does not work in the drilldown to a module.
2017-01-17 MOD-296 The extendable tab XML generator REST endpoint is located in DA-ITSI-OS instead of in common components where it can be used by all modules.
2017-01-17 MOD-591 ITSI displays a misleading error message when a KPI template contains a field that cannot be resolved.
2017-01-17 MOD-498 There is no upper limit to the number of characters a KPI title or description can contain. Long strings can negatively affect performance.
2017-01-17 MOD-309 The Gruntfile.js included in ITSI modules uses double quotes instead of single quotes, which does not conform to the standard for all JavaScript files.
2017-04-17 MOD-2002 When you drilldown from the Events tab, an "Invalid earliest_time" error occurs.


Workaround:
Disable drilldown from the Events tab.

2017-01-17 MOD-439 Some modules do not have descriptions for saved searches.

Application Server Module

Publication date Issue number Description
2017-01-27 MOD-492 If you reuse the same panel within a dashboard, the duplicate panel does not display any event data.

Cloud Services Module

There are no known issues for this release.

Database Module

Publication date Issue number Description
2017-01-17 MOD-586 When a lookup is not configured for TA-Microsoft-SqlServer, ITSI displays a misleading error message on the server drilldown page.

End User Experience Module

There are no known issues for this release.

Load Balancer Module

Publication date Issue number Description
2017-01-27 MOD-492 If you reuse the same panel within a dashboard, the duplicate panel does not display any event data.

Operating System Module

Publication date Issue number Description
2017-04-13 MOD-555 The Storage Free Space % base search runs every minute while the Linux df command runs every 5 minutes. This causes data gaps.
2017-04-10 MOD-1964 Windows data for memory free space is collected at different intervals than the Memory Free % KPI.
2017-01-17 MOD-1398 Line, stack, and area charts do not display a metric gap when no metrics are available during a time period.

Storage Module

There are no known issues for this release.

Virtualization Module

There are no known issues for this release.

Web Server Module

Publication date Issue number Description
2017-03-17 MOD-320 Some KPI ad hoc searches transform data with the stats command and do not retain time fields. The KPIs do not render anything and do not show thresholding details.
2017-03-17 MOD-538 When you add a new tab with panels and refresh the page, the page breaks.
Last modified on 12 May, 2022
PREVIOUS
Fixed issues in Splunk IT Service Intelligence 		  NEXT
Removed features in Splunk IT Service Intelligence

This documentation applies to the following versions of Splunk® IT Service Intelligence: 4.11.3

Was this documentation topic helpful?


You must be logged into splunk.com in order to post comments. Log in now.

Please try to keep this discussion focused on the content covered in this documentation topic. If you have a more general question about Splunk functionality or are experiencing a difficulty with Splunk, consider posting a question to Splunkbase Answers.

0 out of 1000 Characters