Splunk® IT Service Intelligence

Release Notes

This documentation does not apply to the most recent version of Splunk® IT Service Intelligence. For documentation on the most recent version, go to the latest release.

Known issues in Splunk IT Service Intelligence

IT Service Intelligence (ITSI) version 4.13.1 has the following known issues and workarounds.

Adaptive Thresholding

Date filed Issue number Description
2023-04-26 ITSI-29672 KPI preview fails to render sometimes

Workaround:
NA
2022-08-23 ITSI-25903 Threshold Template Sync Fails with Empty Alert Values in threshold template

Backup/Restore and Migration Issues

Date filed Issue number Description
2023-02-28 ITSI-28926 kvstore_to_json.py restore operations do not remove existing services
2022-09-15 ITSI-26204 ITSI Default Scheduled Backup taking hours to complete after upgrade to 4.11.5 (it used to be minutes)

Workaround:
* Run the below curl command to delete the entry in the collection Template:Itsi migration status

{noformat}curl -ku admin https://localhost:8089/servicesNS/nobody/SA-ITOA/storage/collections/data/itsi_migration_status -X DELETE{noformat}

Notable Events

Date filed Issue number Description
2023-06-29 ITSI-31192 All Events tab does not render default columns if they are not present in NEAP JSON definition

Workaround:
# Use the latest ITSI Backup file to edit the NEAP JSON definition and remove the property Template:All events columns and restore the backup.
  1. Go to Episode Review page and add back all the desired columns
2023-03-14 ITSI-29095 Episode Detail Dashboard does not show updated token values
2023-02-13 ITSI-28794 Events are not aggregated into a single episode and split into multiple episodes when the events occur within a small time range.

Workaround:
If you are using correlation searches to generate notable events, make the following changes:
  1. Edit $SPLUNK_HOME/etc/apps/SA-ITOA/local/itsi_rules_engine.properties and add this line:
    rules_engine_feature_blacklist = POLICY_EXECUTOR_ASYNC_SUB_ACTORS, POLICY_EXECUTOR_STATE_RECOVERY


  2. Edit $SPLUNK_HOME/etc/apps/SA-ITOA/local/itsi_event_management.conf and add this line: 
    [tracked_alert]
sort_notable_events = 1
  3. Edit $SPLUNK_HOME/etc/apps/SA-ITOA/local/alert_actions.conf and add this line: 
    [itsi_event_generator]
param.is_use_event_time = 1
  4. Restart Splunk.

If you are directly ingesting notable events through the HEC:

  1. Ensure that the events are ingested in a sorted manner according to the timestamp in an ascending order.
  2. Edit $SPLUNK_HOME/etc/apps/SA-ITOA/local/itsi_rules_engine.properties and add this line: rules_engine_feature_blacklist = POLICY_EXECUTOR_ASYNC_SUB_ACTORS, POLICY_EXECUTOR_STATE_RECOVERY
  3. Restart the Rules Engine.
  4. Navigate to *Activity*, then *Jobs*, then set all filters to "*All*", then Search for Template:Label="itsi event grouping", then stop the job. Wait a couple minutes until the Rules Engine is running again.

2023-01-29 ITSI-28231 Episodes disappear after refresh in episode review dashboard

Workaround:
Add 'Services' filter in Episode Review
2023-01-16 ITSI-28046 Alert action configuration UI not loaded in ITSI when the count of alert actions exceed 30

Workaround:
Keep the count of alert actions in the instance below 30
2022-12-06 ITSI-27595 Actions Rules field names in UI not keeping the upper case upon save
2022-06-07 ITSI-24488 Rules engine search fails to start after upgrade to ITSI 4.13.0

Workaround:
Move the jackson-core-2.10.0.jar and jackson-annotations-2.10.0.jar to the .bkup folder under $SPLUNK_HOME/etc/apps/SA-ITOA/lib/java/event_management/libs directory.

Notable Event Aggregation Policies

Date filed Issue number Description
2023-06-29 ITSI-31192 All Events tab does not render default columns if they are not present in NEAP JSON definition

Workaround:
# Use the latest ITSI Backup file to edit the NEAP JSON definition and remove the property Template:All events columns and restore the backup.
  1. Go to Episode Review page and add back all the desired columns
2023-03-14 ITSI-29095 Episode Detail Dashboard does not show updated token values
2023-02-13 ITSI-28794 Events are not aggregated into a single episode and split into multiple episodes when the events occur within a small time range.

Workaround:
If you are using correlation searches to generate notable events, make the following changes:
  1. Edit $SPLUNK_HOME/etc/apps/SA-ITOA/local/itsi_rules_engine.properties and add this line:
    rules_engine_feature_blacklist = POLICY_EXECUTOR_ASYNC_SUB_ACTORS, POLICY_EXECUTOR_STATE_RECOVERY


  2. Edit $SPLUNK_HOME/etc/apps/SA-ITOA/local/itsi_event_management.conf and add this line: 
    [tracked_alert]
sort_notable_events = 1
  3. Edit $SPLUNK_HOME/etc/apps/SA-ITOA/local/alert_actions.conf and add this line: 
    [itsi_event_generator]
param.is_use_event_time = 1
  4. Restart Splunk.

If you are directly ingesting notable events through the HEC:

  1. Ensure that the events are ingested in a sorted manner according to the timestamp in an ascending order.
  2. Edit $SPLUNK_HOME/etc/apps/SA-ITOA/local/itsi_rules_engine.properties and add this line: rules_engine_feature_blacklist = POLICY_EXECUTOR_ASYNC_SUB_ACTORS, POLICY_EXECUTOR_STATE_RECOVERY
  3. Restart the Rules Engine.
  4. Navigate to *Activity*, then *Jobs*, then set all filters to "*All*", then Search for Template:Label="itsi event grouping", then stop the job. Wait a couple minutes until the Rules Engine is running again.

2023-01-29 ITSI-28231 Episodes disappear after refresh in episode review dashboard

Workaround:
Add 'Services' filter in Episode Review
2023-01-16 ITSI-28046 Alert action configuration UI not loaded in ITSI when the count of alert actions exceed 30

Workaround:
Keep the count of alert actions in the instance below 30
2022-12-06 ITSI-27595 Actions Rules field names in UI not keeping the upper case upon save
2022-06-07 ITSI-24488 Rules engine search fails to start after upgrade to ITSI 4.13.0

Workaround:
Move the jackson-core-2.10.0.jar and jackson-annotations-2.10.0.jar to the .bkup folder under $SPLUNK_HOME/etc/apps/SA-ITOA/lib/java/event_management/libs directory.

Glass Table

Date filed Issue number Description
2022-07-29 ITSI-25262 Font size adjustments and drilldowns for text are not working properly for glass tables after upgrading to ITSI 4.13.1

Workaround:
Issue 1: The font size is not adjustable.

Font size can be adjusted in splunk.markdown at some level with use of the H button from the UI.

# Heading level 1 ## Heading level 2 ### Heading level 3

Issue 2: Drilldown is not supported.

A custom URL can be used in splunk.markdown in place of the drilldown.

KPI Base Searches

Date filed Issue number Description
2023-02-10 ITSI-28784 ITSI warning icon is shown with the message "No valid base searches defined - cigna

Workaround:
N/A
2022-10-05 ITSI-26497 app/itsi/kpi_base_searches_lister error

Workaround:
N/A
2022-08-23 ITSI-25903 Threshold Template Sync Fails with Empty Alert Values in threshold template
2022-07-18 ITSI-25037 'Add Metric' option not working for metric type search in KPI Base search creation

KPI Search Calculation

Date filed Issue number Description
2022-12-16 ITSI-27721 KPI title surrounded with double quotes throws an error while running a KPI Generated Search
2022-09-13 ITSI-26151 ITSI upgrade from 4.9.6 to 4.13.1 causes excessive CPU utilization and skipped searches.

Workaround:
Use the following curl command to delete all the objects from the itsi_kpi_state_cache collection that don't reference any KPIs in your system:

{{curl -k -u {USERNAME} -X DELETE -G "https://{ITSI_SEARCH_HEAD_URI}:{MANAGEMENT_PORT}/servicesNS/nobody/SA-ITOA/storage/collections/data/itsi_kpi_state_cache" -d 'query={"$and":[{"_version":"VERSION NUMBER"},{"_key":{"$ne":"OBJECT ID"}},]}&fields=_key,_version'}}

  • Replace {{{USERNAME}}}, {{{ITSI_SEARCH_HEAD_URI}}} and {{{MANAGEMENT_PORT}}} with appropriate values before running the command.
  • Determine the ITSI versions you want to delete objects from the itsi_kpi_state_cache collection and replace "VERSION NUMBER" with the version(s).
  • Determine the object IDs of objects you want to keep for a specific version and replace "OBJECT ID" with this value.
2022-04-28 ITSI-23284 Deleted KPI lanes still showing in deep dive when the URL is refreshed.

Performance

Date filed Issue number Description
2023-04-26 ITSI-29672 KPI preview fails to render sometimes

Workaround:
NA
2022-09-13 ITSI-26151 ITSI upgrade from 4.9.6 to 4.13.1 causes excessive CPU utilization and skipped searches.

Workaround:
Use the following curl command to delete all the objects from the itsi_kpi_state_cache collection that don't reference any KPIs in your system:

{{curl -k -u {USERNAME} -X DELETE -G "https://{ITSI_SEARCH_HEAD_URI}:{MANAGEMENT_PORT}/servicesNS/nobody/SA-ITOA/storage/collections/data/itsi_kpi_state_cache" -d 'query={"$and":[{"_version":"VERSION NUMBER"},{"_key":{"$ne":"OBJECT ID"}},]}&fields=_key,_version'}}

  • Replace {{{USERNAME}}}, {{{ITSI_SEARCH_HEAD_URI}}} and {{{MANAGEMENT_PORT}}} with appropriate values before running the command.
  • Determine the ITSI versions you want to delete objects from the itsi_kpi_state_cache collection and replace "VERSION NUMBER" with the version(s).
  • Determine the object IDs of objects you want to keep for a specific version and replace "OBJECT ID" with this value.

Role Based Access Controls

Date filed Issue number Description
2022-07-13 ITSI-24979 In Alerts and Episodes, users can view and access all saved episode review pages in 'Show Alternate Views' collapsible panel and can also delete any view of other users that is private
2021-12-14 ITSI-20605, ITSI-22366 Occasionally after ITSI upgrade, non-admin users get Oops Page - local.meta corrupted during the upgrade

Workaround:
Clean up all permissions on ITSI views in itsi/metadata/local.meta (and sync on SHC)

The workaround is to clean up the stanza in local.meta on the all the SH. remove all the stanza like \[views/....]  that have no valid access settings, (access = delete : \[  ], read : \[  ], write : \[  ]) and that are not custom views from your users.

As they may be many, to confirm, you can compare to the list in default.meta And you also can look at the modtime field in the stanza, as they are probably all identical.

Service Analyzer

Date filed Issue number Description
2023-02-17 ITSI-28826 Changes to health score color values in threshold_labels.conf do not appear in the service analyzer.

Service Definition

Date filed Issue number Description
2022-10-11 ITSI-26591 Thresholding radio button selecting "Use Thresholding Template" instead of "Set Custom Thresholds"

Service Templates

Date filed Issue number Description
2023-02-10 ITSI-28784 ITSI warning icon is shown with the message "No valid base searches defined - cigna

Workaround:
N/A
2022-08-23 ITSI-25903 Threshold Template Sync Fails with Empty Alert Values in threshold template

Uncategorized issues

Date filed Issue number Description
2023-03-20 ITSI-29133 Episode Review dashboard panel for Noise reduction should not show "Missing property: majorValue"
2023-03-15 ITSI-29099 Intermitently facing the issue where episodes status does not reflect the NEAP when the Episdoes are being closed
2023-01-17 ITSI-28054 FIPS Enabled on RHEL-8 with ITSI 4.13.1 and JDK 8 throws KV Store initialisation error
2023-01-09 ITSI-27961 Bidirectional Ticketing Correlation Search hits "subsearch limit of 50000 reached" when the collection itsi_notable_event_ticketing has more than 50000 entries

Workaround:
# Navigate to ITSI -> Configuration -> Correlation Searches
  1. Click on Bidirectional Ticketing
  2. Paste the following search in the Search field and then click on Save. Also enable the CS if it has been disabled

{noformat}| datamodel Ticket_Management Incident search | rename All_Ticket_Management.ticket_id as ticket_id | join ticket_id [search sourcetype="snow:incident" index="<snow_index>" | where _indextime > now() - <max_lookback_time>] | lookup itsi_notable_event_external_ticket tickets.ticket_id as ticket_id OUTPUTNEW tickets.ticket_system event_id | where isnotnull(event_id) | rename tickets.* as * | eventstats values(event_id) as group_id last(ticket_system) as ticket_system by ticket_id | fields - dv_* | table * | makemv group_id | mvexpand group_id | eval bidirectional_ticketing=1, snow_hash = number + "!" + group_id + "!" + sys_updated_on | search NOT [| search index="itsi_tracked_alerts" | fields snow_hash] | dedup snow_hash{noformat}

Change the placeholders {{<snow_index>}} and {{<max_lookback_time>}} in the above search with values according to the customer's requirements

2022-12-20 ITSI-27741 When closing episodes in bulk, episodes with different statuses display as closed but aren't actually closed.

Workaround:
During the bulk update of the episodes from the UI, make sure that all the Episodes selected for the bulk update at a time have same Status.
2022-12-08 ITSI-27617 Anomaly detection configuration fails with error "Error in 'naccum' command"

Workaround:
Move the directory mad_lib from "./{color:#bf2600}*etc/apps/SA-ITSI-MetricAD/bin/"*{color} to a directory outside the splunk
2022-12-06 ITSI-27586 EA Smart Recycling in retention policy not considering all end status in case of custom configurations
2022-11-22 ITSI-27450, ITSI-27449, ITSI-27451 is_partial_data=0 is not working as Documented for maintenance_services_interface/<object_type>/<_key>

Workaround:
Use Template:Is partial data= to use it as Template:Is partial data=0
2022-11-22 ITSI-27449, ITSI-27450 The is_partial_data=0 is not working as Documented for itoa_interface/<object_type>/<_key> POST call

Workaround:
Use Template:Is partial data= to use it as Template:Is partial data=0
2022-10-13 ITSI-26687 Vital metric sorting has a small caveat while filtering with entity Dimension filter on the Infrastructure overview page
2022-10-11 ITSI-26585 Entities status is getting "Unstable" from "Active" when installing SA4CP 1.7.0 with ITEW

Workaround:
# Go to Settings → Searches, Reports, and Alerts
  1. Select App → Content Pack for ITSI Monitoring and Alerting (DA-ITSI-CP-monitoring-alerting)
  2. Select Owner → All
  3. Click on the saved search → ITSI Import Objects - itsi_entity_name_normalizer
  4. Edit the search to → | inputlookup itsi_entities | eval entity_name=title
  5. Save the saved search
2022-09-19 ITSI-26224 Importing large numbers of entities at once causes a 414 HTTP error

Workaround:
To work around the core issue we split the objects to be imported into two searches with roughly half the results and this is working. We are unsure where the limit is where this will fail again.
2022-08-22 ITSI-25886 Fix BDT event back filling due to not indexed into grouped alert
2022-07-12 ITSI-24964 ITSI Searches ("Date Range", "Date & Time Range") do not honor auto-generated values; new Real-time search option fails (tstats not supported in a real-time search)

Workaround:
For Date & Time selections: manually enter/replace any portion of the auto-filled date for both start and end dates (even if replacing with the same value); or, select date from the calendar dropdown.

For Date selection only: No workaround found so far.

2021-09-01 ITSI-18709 ITSI redirects to suite_redirect 500 Internal Server Error - because of python library isolation between apps

Workaround:
Step 1: Identify all the splunklib directories within the splunk apps directory using command find . -name 'splunklib' | xargs -r ls -lah.

Step 2: For each directory listed in step 1, check if file six.py is present.

Step 3: Copy the six.py from an existing splunklib directory into all the missing directories.

Step 4: Clean the cached files using find . -name "*.pyc" -delete

Step 5: Restart Splunk on the ITE Work or ITSI search head.

2019-05-30 ITSI-3322 If you add a correlation search in ITSI which contains a sub-search returning into an eval, you get a message "Invalid search string: This search cannot be parsed when parse_only is set to true."

Workaround:
You can't use a sub-search returning into an eval in a correlation search. As a workaround, create and save a basic correlation search with all of the information you want outside of the search. Then as an admin user, go to Settings > Searches, reports, and alerts and open the correlation search you just created. Add the sub-search you were trying to add there.
Last modified on 02 March, 2024
Fixed issues in Splunk IT Service Intelligence   Removed features in Splunk IT Service Intelligence

This documentation applies to the following versions of Splunk® IT Service Intelligence: 4.13.1

Acrobat logo Download manual
Acrobat logo Download this page

Was this topic useful?







You must be logged into splunk.com in order to post comments. Log in now.

Please try to keep this discussion focused on the content covered in this documentation topic. If you have a more general question about Splunk functionality or are experiencing a difficulty with Splunk, consider posting a question to Splunkbase Answers.

0 out of 1000 Characters