Splunk® IT Service Intelligence

Release Notes

This documentation does not apply to the most recent version of Splunk® IT Service Intelligence. For documentation on the most recent version, go to the latest release.

Known issues in Splunk IT Service Intelligence

This version of IT Service Intelligence (ITSI) has the following known issues and workarounds.

Highlighted issues

Date filed Issue number Description
2023-05-06 ITSI-30026 Event generated from Provider are not getting grouped on Federated Search head

Workaround:
Event generated from provider gets grouped through the rule engine periodic backfill.
2022-09-07 ITSI-26097 Entities and vital metrics are not populating on federated search setup

Workaround:
# Workaround for enabling entity discovery with federated search setup:

Change |makeresults to |makeresults | head 1 in saved searches from SPLUNK_HOME/etc/apps/itsi/(default) and (local)/savedsearch.conf.

Alternatively, you can make the change through the IT Service Intelligence interface by selecting *Settings > Searches, reports and alerts*, and then searching for the saved search name on the Searches, Reports, and Alerts page. For example, to discover *nix entities, you need go to '*ITSI Import Object - OS*' and revise '|*makeresults*' to become '|makeresults | head 1'

2. Workaround to ensure that vital metrics populate with federated search setup:

a. Change '|makeresults' to '|makeresults | head 1' in the following two macros 

i. gen_eval_fields(1)
ii. gen_as_fields(2)

Adaptive Thresholding

Date filed Issue number Description
2023-06-07 ITSI-30577 Values from threshold templates don't appear in Service Definition
2023-03-21 ITSI-29200 Change in the threshold value of a KPI in a service template does not update in the services linked to it.

Workaround:
In the threshold editor, create a new temporary time policy, save, and select *Replace all KPI thresholds.* This will force the propagation of all time policies to all linked services. After this occurs, you can also delete the extra policy.
2023-01-03 ITSI-27867 In Adaptive Thresholding Clicking on apply button shows any warning as errors in UI.

Backup/Restore and Migration Issues

Date filed Issue number Description
2023-10-12 ITSI-32459 Cleanup the migration_helper folder before the restore of the backup starts
2023-04-19 ITSI-29586 Unable to restore default scheduled backup

Workaround:
Download the Default Scheduled Backup and restore the downloaded backup
2023-03-31 ITSI-29305 Restore is failing for the large backup on the WiredTiger Storage engine for the MongoDB
2023-02-28 ITSI-28926 kvstore_to_json.py restore operations do not remove existing services

Bulk Import

Date filed Issue number Description
2023-09-12 ITSI-32028 Error on entity import : 404 Not Found on GET to /servicesNS/nobody/SA-ITOA/storage/collections/data/itsi_import_objects_cache
2023-04-12 ITSI-29489 module "ITSI Operating System" or other modules missing from entity import list options - when the SH has more than 30 DA-ITSI* apps installed

Entities

Date filed Issue number Description
2023-08-14 ITSI-31723 Error modal appears when user attempts to filter entities with a parenthesis in the name on entity management page

Workaround:
Use backslash before the special character. To search for "myhost(" try "myhost\("
2023-04-19 ITSI-29586 Unable to restore default scheduled backup

Workaround:
Download the Default Scheduled Backup and restore the downloaded backup
2022-09-07 ITSI-26097 Entities and vital metrics are not populating on federated search setup

Workaround:
# Workaround for enabling entity discovery with federated search setup:

Change |makeresults to |makeresults | head 1 in saved searches from SPLUNK_HOME/etc/apps/itsi/(default) and (local)/savedsearch.conf.

Alternatively, you can make the change through the IT Service Intelligence interface by selecting *Settings > Searches, reports and alerts*, and then searching for the saved search name on the Searches, Reports, and Alerts page. For example, to discover *nix entities, you need go to '*ITSI Import Object - OS*' and revise '|*makeresults*' to become '|makeresults | head 1'

2. Workaround to ensure that vital metrics populate with federated search setup:

a. Change '|makeresults' to '|makeresults | head 1' in the following two macros 

i. gen_eval_fields(1)
ii. gen_as_fields(2)

Entity Rules

Date filed Issue number Description
2023-02-23 ITSI-28871 Entity filter rule considering empty value as a wildcard (*)

Notable Events

Date filed Issue number Description
2024-02-08 ITSI-34393 BDT event should not satisfy 'if episode is broken' action rule for inactive episodes
2023-06-29 ITSI-31192 All Events tab does not render default columns if they are not present in NEAP JSON definition

Workaround:
# Use the latest ITSI Backup file to edit the NEAP JSON definition and remove the property Template:All events columns and restore the backup.
  1. Go to Episode Review page and add back all the desired columns
2023-06-19 ITSI-31057 host field value not visible to Rules Engine
2023-06-02 ITSI-30500 NEAP filtering criteria with value *(wildcard) does not satisfy the events which contain \n(line break) in the value

Workaround:
Add another negative filtering criteria for the field.

For example, if we have added a filtering criteria Source Matches * then event will not be picked up by a custom NEAP. You need to add another negative filtering criteria with the or condition so it will be similar to Source Does not Match *.

2023-05-12 ITSI-30099 When multiple actions are triggered, field does not get updated according to the last action rule
2023-05-06 ITSI-30026 Event generated from Provider are not getting grouped on Federated Search head

Workaround:
Event generated from provider gets grouped through the rule engine periodic backfill.
2023-03-23 ITSI-29214 Episode Detail Panel does not display full name of user correctly
2023-03-15 ITSI-29098 Re-ordering the columns does not work in Episode review
2023-02-08 ITSI-28707 Color for custom severity is not displayed correctly in Correlation Search Builder, Notable Event Aggregation Policy Editor and Episode Review page
2023-01-12 ITSI-28015 The episode link in "Share Episode" does not get updated in right click menu
2022-11-04 ITSI-27028 When Identifier Fields are specified for Notables and Smart Mode is enabled, the Episodes do not show the identifier fields

Notable Event Aggregation Policies

Date filed Issue number Description
2024-02-08 ITSI-34393 BDT event should not satisfy 'if episode is broken' action rule for inactive episodes
2023-06-29 ITSI-31192 All Events tab does not render default columns if they are not present in NEAP JSON definition

Workaround:
# Use the latest ITSI Backup file to edit the NEAP JSON definition and remove the property Template:All events columns and restore the backup.
  1. Go to Episode Review page and add back all the desired columns
2023-06-19 ITSI-31057 host field value not visible to Rules Engine
2023-06-02 ITSI-30500 NEAP filtering criteria with value *(wildcard) does not satisfy the events which contain \n(line break) in the value

Workaround:
Add another negative filtering criteria for the field.

For example, if we have added a filtering criteria Source Matches * then event will not be picked up by a custom NEAP. You need to add another negative filtering criteria with the or condition so it will be similar to Source Does not Match *.

2023-05-12 ITSI-30099 When multiple actions are triggered, field does not get updated according to the last action rule
2023-05-06 ITSI-30026 Event generated from Provider are not getting grouped on Federated Search head

Workaround:
Event generated from provider gets grouped through the rule engine periodic backfill.
2023-03-23 ITSI-29214 Episode Detail Panel does not display full name of user correctly
2023-03-15 ITSI-29098 Re-ordering the columns does not work in Episode review
2023-02-08 ITSI-28707 Color for custom severity is not displayed correctly in Correlation Search Builder, Notable Event Aggregation Policy Editor and Episode Review page
2023-01-12 ITSI-28015 The episode link in "Share Episode" does not get updated in right click menu
2022-11-04 ITSI-27028 When Identifier Fields are specified for Notables and Smart Mode is enabled, the Episodes do not show the identifier fields

Glass Table

Date filed Issue number Description
2023-04-11 ITSI-29450 variable should be pass between different widgets in glass table
2023-03-26 ITSI-29246 transurban text is overlapped on Glass Table and unreadable since upgrade to 4.16.0

Workaround:
The Workaround is to convert Template:Splunk.markdown to Template:Viz.text field in the glasstable.
2023-01-23 ITSI-28150 Error in console when user add KPI/ad hoc search for the splunk.singlevalueicon
2023-01-10 ITSI-27969 Ad hoc search should work properly even if we add it after deleting the existing the kpi data source from the visualization

Workaround:
Remove the value of options field from glass table source code in visualization when you delete the KPI data source and add adhoc data source in same visualization.
2023-01-05 ITSI-27888 Move Forward/Backward is not working when initially add visualization to a black canvas
2023-01-05 ITSI-27886 splunk.markdown adds unexpected background colour and text colour when leading spaces are used in text
2023-01-05 ITSI-27882 Error in console if splunk.ellipse and splunk.rectangle are present
2021-12-17 ITSI-20748 Service Swapping weirdness on Glass Table

KPI Base Searches

Date filed Issue number Description
2023-06-20 ITSI-31085 KPI Backfill searches run under 'Search' app context instead of ITSI/SA-ITOA app context
2023-02-23 ITSI-28869 Adhoc searches should not be validated while creating KPI base with metrics search option

KPI Search Calculation

Date filed Issue number Description
2023-06-20 ITSI-31085 KPI Backfill searches run under 'Search' app context instead of ITSI/SA-ITOA app context
2023-06-06 ITSI-30550 Show search/parser errors in UI for KPI creation model
2023-02-24 ITSI-28886 mod_time and retirable appear as a metric_name in itsi_summary_metrics and unnecessarily creates extra datapoints

Maintenance Window

Date filed Issue number Description
2023-03-12 ITSI-29078 Retired Entities not being filtered out of Maintenance Window List of Entities

Workaround:
N/A

Role Based Access Controls

Date filed Issue number Description
2023-05-04 ITSI-30017 A user in itoa_user role cannot open ITSI homeview in SHC.

Workaround:
We have to add the list_search_head_clustering capability to the default authorize.conf.

Service Analyzer

Date filed Issue number Description
2023-06-09 ITSI-30822 ITSI degraded-entities-search-manager may have caused indexers cluster to crash

Workaround:
If Service Analyser is running for more than 1 week's time range and search is going through millions of events try to limit the service analyser time range to less than 1 week to limit the search time range.
2023-06-07 ITSI-30580 When the dbconnect app is installed, non-admin ITSI users cannot access their homepage but are routed to the upgrade page.

Workaround:
Add the db_connect_read_app_conf capability to the custom user with a non-admin role. Enable this capability in the default authorize.conf file.
2023-04-14 ITSI-29512 Service Analyzer tiles have service title in non-uniform text color
2023-02-17 ITSI-28826 Changes to health score color values in threshold_labels.conf do not appear in the service analyzer.

Service Definition

Date filed Issue number Description
2023-05-29 ITSI-30378 Time policy auto-shifts by one hour upon user addition

Workaround:
Shifts time block by using kvstore_to_json.py script mode 3

Service Templates

Date filed Issue number Description
2023-08-25 ITSI-31867 Entity rule fields are no longer in view only mode for the OOTB service templates
2023-03-21 ITSI-29200 Change in the threshold value of a KPI in a service template does not update in the services linked to it.

Workaround:
In the threshold editor, create a new temporary time policy, save, and select *Replace all KPI thresholds.* This will force the propagation of all time policies to all linked services. After this occurs, you can also delete the extra policy.
2022-10-18 ITSI-26757 Refresh queue is overriding base service template object while linking the service to it for more than 15 concurrent service creation.

Uncategorized issues

Date filed Issue number Description
2024-02-29 ITSI-34551 Breaking event does not trigger breaking action rules
2024-01-23 ITSI-34041 ITSI Episode view triggers a search to populate linked tickets, that is looking back to epoch time=1 second till now
2023-12-25 ITSI-33583 Episode review timeline should be updated when the policy filter is applied
2023-12-07 ITSI-33278 Cannot create a correlation search with all special character
2023-11-19 ITSI-33134 When the episode from 2nd page is selected and the table refreshed The focus from the episode is getting lost

Workaround:
The workaround for the issue mentioned here is, adding/updating the Template:Itsi notable group lookup macro from the Template:Etc/apps/SA-ITOA/local/macros.conf file.

Add {{itsi_policy_id | eval policy_id=itsi_policy_id, _itsi_is_group_broken=if(is_active==0,1,0)}} at the end of the definition of the Template:Itsi notable group lookup macro so that it would look like below.

{noformat}[itsi_notable_group_lookup] args = definition = lookup itsi_notable_group_user_lookup _key AS itsi_group_id OUTPUT owner severity status instruction | lookup itsi_notable_group_system_lookup _key AS itsi_group_id OUTPUT title description start_time last_time is_active event_count itsi_policy_id | eval policy_id=itsi_policy_id, _itsi_is_group_broken=if(is_active==0,1,0){noformat}

This will make sure that the episode focus will not be lost for another 2 filters Template:Policy and Template:Show Episodes . So, for the filters Status, Owner, Severity, Policy and Show Episodes the episode focus will not be lost.

2023-11-15 ITSI-33113 Bulk Acknowledge Episode can be executed for already Acknowledged episode by another user
2023-10-17 ITSI-32621 Notable drildown search not working on Episode view with sourcetype and colon
2023-09-05 ITSI-31978 Correlation search edit page malfunctions when time range set to "All Time"

Workaround:
*Workaround 1*
  1. From Setting-->Searches, Reports, and Alerts-->search for CS-->Edit Alert-->Update time range to Template:Last 15 Minutes from Template:All Time
  • Workaround 2*
  1. Update/add itsi/local/savedsearches.conf as following:

{noformat}dispatch.earliest_time = 0 dispatch.latest_time = now{noformat}

2023-08-18 ITSI-31763 Closing events not excluded in periodic backfill search
2023-08-16 ITSI-31748 Unable to edit cron expression field for the ELM policy
2023-08-11 ITSI-31708 KPI backfilling is not working properly for services linked with more number of entities
2023-07-22 ITSI-31402 Running kvstore_to_json.py in cloud fails, throws exception about invalid syntax

Workaround:
Run the kvstore_to_json.py with python 3

{noformat}bin/splunk cmd python3 etc/apps/SA-ITOA/bin/kvstore_to_json.py -s 8089 -u admin -m 1{noformat}

2023-06-29 ITSI-31185 Entity status inconsistency - Infrastructure Overview

Workaround:
Will provide the script to the customer to delete the entities from the itsi_services and itsi_bulk_import_status_cache collection.
2023-05-24 ITSI-30342 Restoring recurring and currently applicable custom threshold window is not getting activated
2023-05-11 ITSI-30095 Change all non-macro based OOTB references to ITSI indexes to use macros
2023-05-11 ITSI-30097, ITSI-32375 Status chart on entity type health page should display all the appropriate statuses
2023-05-10 ITSI-30068 Event Analytics Monitoring Rules Engine Information panel uses an All time search

Workaround:
  • Edit the Event Analytics Monitoring Dashboard.
  • Click on the magnifying glass under the Rules Engine Information panel.
  • Change the Time Range to Shared Time Picker (time_token).

    • 2023-04-30 ITSI-29738 Clicking on "Run fix" button is not resolving the issue of "Missing KPI base search"
    2023-04-20 ITSI-29608 itsi_bidirectional_ticking macro should use macros for index and source types being used within the search
    2023-04-20 ITSI-29609, ITSI-30886 itsi_bmc_bidirectional_ticking macro should use macros for index and source types being used within the search
    2023-04-17 ITSI-29521 Fix the itsi_module_interface API to fetch more than 30 apps having prefix DA-ITSI
    2023-04-13 ITSI-29506 App inspect report failing for authored content pack with saved searches

    Workaround:
    Please follow the below steps.
    1. Unzip the custom content pack.
    2. Remove the "action.script" key from each of the saved searches located in the file "./default/savedsearches.conf" and save your changes.
    3. Zip the custom content pack and use it.
    2023-03-20 ITSI-29133 Episode Review dashboard panel for Noise reduction should not show "Missing property: majorValue"
    2023-02-17 ITSI-28829 The timebased breaking event replaces the episode information fields.
    2023-02-13 ITSI-28799 Content Authorship does not support custom font size in splunk.markdown feature of glass table

    Workaround:
    Due to a defect in the packaging of Content Authorship, the custom font size is retained, but the option that we have to prioritise the custom font size over the Default value is not retained. Please follow the below steps to get the custom font size for the glass tables which are packaged using Content Authorship.
    1. Go to the Edit section of Glass table
    2. Select the text ( splunk.markdown object)
    3. In the Configuration Pane (right side), click on the "General" section
    4. Change Font size from "Default" to "Custom"
    5. Save your changes
    2023-01-12 ITSI-28026 "Show Alternative Views" UI toggle too small
    2023-01-09 ITSI-27961 Bidirectional Ticketing Correlation Search hits "subsearch limit of 50000 reached" when the collection itsi_notable_event_ticketing has more than 50000 entries

    Workaround:
    # Navigate to ITSI -> Configuration -> Correlation Searches
    1. Click on Bidirectional Ticketing
    2. Paste the following search in the Search field and then click on Save. Also enable the CS if it has been disabled

    {noformat}| datamodel Ticket_Management Incident search | rename All_Ticket_Management.ticket_id as ticket_id | join ticket_id [search sourcetype="snow:incident" index="<snow_index>" | where _indextime > now() - <max_lookback_time>] | lookup itsi_notable_event_external_ticket tickets.ticket_id as ticket_id OUTPUTNEW tickets.ticket_system event_id | where isnotnull(event_id) | rename tickets.* as * | eventstats values(event_id) as group_id last(ticket_system) as ticket_system by ticket_id | fields - dv_* | table * | makemv group_id | mvexpand group_id | eval bidirectional_ticketing=1, snow_hash = number + "!" + group_id + "!" + sys_updated_on | search NOT [| search index="itsi_tracked_alerts" | fields snow_hash] | dedup snow_hash{noformat}

    Change the placeholders {{<snow_index>}} and {{<max_lookback_time>}} in the above search with values according to the customer's requirements

    2023-01-06 ITSI-27928, ITSI-27925 Private Episodes should be created or read even if the capabilities are not provided
    2023-01-02 ITSI-27863 Vital metrics show up as N/A for individual entities in Entity Overview Page

    Workaround:
    The macro has the id field which is getting used in all entity types and saved searches. Remove the id field from the vital metrics SPL and split by fields, or rename this field.
    2022-12-27 ITSI-27834, ITSI-27835 The Splunk Enterprise Objects filter for Owner does not have the option of "nobody" during custom content pack creation
    2022-12-08 ITSI-27627 Correlation search - count API throws "500 internal server error" when filter is performed on the name which doesn't match with any search
    2022-11-28 ITSI-27815 Invalid token for Auto Generated ITSI Event Management Token in Noah stacks

    Workaround:
    # Navigate to Settings → Data inputs → HTTP Event Collector
    1. For the input Template:Auto Generated ITSI Event Management Token, click on "Delete". This will delete the duplicate token value and within few seconds it will be replaced by the correct one
    2022-09-19 ITSI-26219, ITSI-26290 Support of splunk.choropleth.svg is missing in ITSI partial backup, Content authorship and itsimodels
    2021-08-25 ITSI-18574 Base search "head" command is removing relevant event data for snapshots, probably depending on the time you look at it

    All ITSI Modules

    Publication date Issue number Description
    2017-03-21 ITOA-7585 When you bulk add services and an error caused by the racing condition occurs, the incorrect message "itsi_module does not exist" is displayed.
    2017-03-07 MOD-979 KPIs do not have consistent backfill settings across all modules.
    2017-01-17 MOD-452 The Analyze KPI button on the Service Details page is broken.
    2017-01-17 MOD-402 The Export to PDF option does not work in the drilldown to a module.
    2017-01-17 MOD-296 The extendable tab XML generator REST endpoint is located in DA-ITSI-OS instead of in common components where it can be used by all modules.
    2017-01-17 MOD-591 ITSI displays a misleading error message when a KPI template contains a field that cannot be resolved.
    2017-01-17 MOD-498 There is no upper limit to the number of characters a KPI title or description can contain. Long strings can negatively affect performance.
    2017-01-17 MOD-309 The Gruntfile.js included in ITSI modules uses double quotes instead of single quotes, which does not conform to the standard for all JavaScript files.
    2017-04-17 MOD-2002 When you drilldown from the Events tab, an "Invalid earliest_time" error occurs.


    Workaround:
    Disable drilldown from the Events tab.

    2017-01-17 MOD-439 Some modules do not have descriptions for saved searches.

    Application Server Module

    Publication date Issue number Description
    2017-01-27 MOD-492 If you reuse the same panel within a dashboard, the duplicate panel does not display any event data.

    Cloud Services Module

    There are no known issues for this release.

    Database Module

    Publication date Issue number Description
    2017-01-17 MOD-586 When a lookup is not configured for TA-Microsoft-SqlServer, ITSI displays a misleading error message on the server drilldown page.

    End User Experience Module

    There are no known issues for this release.

    Load Balancer Module

    Publication date Issue number Description
    2017-01-27 MOD-492 If you reuse the same panel within a dashboard, the duplicate panel does not display any event data.

    Operating System Module

    Publication date Issue number Description
    2017-04-13 MOD-555 The Storage Free Space % base search runs every minute while the Linux df command runs every 5 minutes. This causes data gaps.
    2017-04-10 MOD-1964 Windows data for memory free space is collected at different intervals than the Memory Free % KPI.
    2017-01-17 MOD-1398 Line, stack, and area charts do not display a metric gap when no metrics are available during a time period.

    Storage Module

    There are no known issues for this release.

    Virtualization Module

    There are no known issues for this release.

    Web Server Module

    Publication date Issue number Description
    2017-03-17 MOD-320 Some KPI ad hoc searches transform data with the stats command and do not retain time fields. The KPIs do not render anything and do not show thresholding details.
    2017-03-17 MOD-538 When you add a new tab with panels and refresh the page, the page breaks.
    Last modified on 27 March, 2024
    Fixed issues in Splunk IT Service Intelligence   Removed features in Splunk IT Service Intelligence

    This documentation applies to the following versions of Splunk® IT Service Intelligence: 4.16.0 Cloud only

    Acrobat logo Download manual
    Acrobat logo Download this page

    Please expect delayed responses to documentation feedback while the team migrates content to a new system. We value your input and thank you for your patience as we work to provide you with an improved content experience!

    Was this topic useful?







    You must be logged into splunk.com in order to post comments. Log in now.

    Please try to keep this discussion focused on the content covered in this documentation topic. If you have a more general question about Splunk functionality or are experiencing a difficulty with Splunk, consider posting a question to Splunkbase Answers.

    0 out of 1000 Characters