Splunk® IT Service Intelligence

Release Notes

Acrobat logo Download manual as PDF


This documentation does not apply to the most recent version of Splunk® IT Service Intelligence. For documentation on the most recent version, go to the latest release.
Acrobat logo Download topic as PDF

Known issues in Splunk IT Service Intelligence

This version of IT Service Intelligence (ITSI) has the following known issues and workarounds.

Highlighted issues

Date filed Issue number Description
2023-05-06 ITSI-30026 Event generated from Provider are not getting grouped on Federated Search head

Workaround:
Event generated from provider gets grouped through the rule engine periodic backfill.
2022-09-07 ITSI-26097 Entities and vital metrics are not populating on federated search setup

Workaround:
# Workaround for enabling entity discovery with federated search setup:

Change |makeresults to |makeresults | head 1 in saved searches from SPLUNK_HOME/etc/apps/itsi/(default) and (local)/savedsearch.conf.

Alternatively, you can make the change through the IT Service Intelligence interface by selecting *Settings > Searches, reports and alerts*, and then searching for the saved search name on the Searches, Reports, and Alerts page. For example, to discover *nix entities, you need go to '*ITSI Import Object - OS*' and revise '|*makeresults*' to become '|makeresults | head 1'

2. Workaround to ensure that vital metrics populate with federated search setup:

a. Change '|makeresults' to '|makeresults | head 1' in the following two macros

i. gen_eval_fields(1)
ii. gen_as_fields(2)

Adaptive Thresholding

Date filed Issue number Description
2023-05-16 ITSI-30132 Backport the bug fixes for 4.15.2
2023-04-26 ITSI-29672 KPI preview fails to render sometimes

Workaround:
NA
2023-03-21 ITSI-29200 Change in the threshold value of a KPI in a service template does not update in the services linked to it.

Workaround:
In the threshold editor, create a new temporary time policy, save, and select *Replace all KPI thresholds.* This will force the propagation of all time policies to all linked services. After this occurs, you can also delete the extra policy.
2023-01-03 ITSI-27867 In Adaptive Thresholding Clicking on apply button shows any warning as errors in UI.

Backup/Restore and Migration Issues

Date filed Issue number Description
2023-05-14 ITSI-30105 Backport the fix to accelerate the fields in itsi_service and itsi_base_service_template collection
2023-03-31 ITSI-29305 Restore is failing for the large backup on the WiredTiger Storage engine for the MongoDB
2023-02-28 ITSI-28926 kvstore_to_json.py restore operations do not remove existing services

Bulk Import

Date filed Issue number Description
2022-09-06 ITSI-26033 Bulk import - List of entities takes a long time to load in the preview for service section

Entities

Date filed Issue number Description
2023-08-14 ITSI-31723 Error modal appears when user attempts to filter entities with a parenthesis in the name on entity management page

Workaround:
Use backslash before the special character. To search for "myhost(" try "myhost\("
2022-09-07 ITSI-26097 Entities and vital metrics are not populating on federated search setup

Workaround:
# Workaround for enabling entity discovery with federated search setup:

Change |makeresults to |makeresults | head 1 in saved searches from SPLUNK_HOME/etc/apps/itsi/(default) and (local)/savedsearch.conf.

Alternatively, you can make the change through the IT Service Intelligence interface by selecting *Settings > Searches, reports and alerts*, and then searching for the saved search name on the Searches, Reports, and Alerts page. For example, to discover *nix entities, you need go to '*ITSI Import Object - OS*' and revise '|*makeresults*' to become '|makeresults | head 1'

2. Workaround to ensure that vital metrics populate with federated search setup:

a. Change '|makeresults' to '|makeresults | head 1' in the following two macros

i. gen_eval_fields(1)
ii. gen_as_fields(2)

Entity Rules

Date filed Issue number Description
2023-02-23 ITSI-28871 Entity filter rule considering empty value as a wildcard (*)

Notable Events

Date filed Issue number Description
2023-06-29 ITSI-31192 All Events tab does not render default columns if they are not present in NEAP JSON definition

Workaround:
# Use the latest ITSI Backup file to edit the NEAP JSON definition and remove the property Template:All events columns and restore the backup.
  1. Go to Episode Review page and add back all the desired columns
2023-06-19 ITSI-31057 host field value not visible to Rules Engine
2023-06-02 ITSI-30500 NEAP filtering criteria with value *(wildcard) does not satisfy the events which contain \n(line break) in the value

Workaround:
Add another negative filtering criteria for the field.

For example, if we have added a filtering criteria Source Matches * then event will not be picked up by a custom NEAP. You need to add another negative filtering criteria with the or condition so it will be similar to Source Does not Match *.

2023-05-06 ITSI-30026 Event generated from Provider are not getting grouped on Federated Search head

Workaround:
Event generated from provider gets grouped through the rule engine periodic backfill.
2023-03-30 ITSI-29292 Update information in Skipped Events panel in Event Analytics Monitoring Dashboard
2023-03-23 ITSI-29214 Episode Detail Panel does not display full name of user correctly
2023-03-14 ITSI-29095 Episode Detail Dashboard does not show updated token values
2023-02-08 ITSI-28707 Color for custom severity is not displayed correctly in Correlation Search Builder, Notable Event Aggregation Policy Editor and Episode Review page
2023-01-16 ITSI-28046 Alert action configuration UI not loaded in ITSI when the count of alert actions exceed 30

Workaround:
Keep the count of alert actions in the instance below 30
2023-01-12 ITSI-28015 The episode link in "Share Episode" does not get updated in right click menu
2022-12-20 ITSI-27751 Episode Review arbitrary search filter with AND & OR conditions fail to match events under certain scenarios

Workaround:
Avoid using brackets (), extra whitespaces, the operator !=, and double quotes "" in the search filter
2022-12-06 ITSI-27595 Actions Rules field names in UI not keeping the upper case upon save
2022-11-04 ITSI-27028 When Identifier Fields are specified for Notables and Smart Mode is enabled, the Episodes do not show the identifier fields
2022-10-25 ITSI-26825 Episode Review timeline search is triggered even when summary dashboard is closed which wastes resources.

Notable Event Aggregation Policies

Date filed Issue number Description
2023-06-29 ITSI-31192 All Events tab does not render default columns if they are not present in NEAP JSON definition

Workaround:
# Use the latest ITSI Backup file to edit the NEAP JSON definition and remove the property Template:All events columns and restore the backup.
  1. Go to Episode Review page and add back all the desired columns
2023-06-19 ITSI-31057 host field value not visible to Rules Engine
2023-06-02 ITSI-30500 NEAP filtering criteria with value *(wildcard) does not satisfy the events which contain \n(line break) in the value

Workaround:
Add another negative filtering criteria for the field.

For example, if we have added a filtering criteria Source Matches * then event will not be picked up by a custom NEAP. You need to add another negative filtering criteria with the or condition so it will be similar to Source Does not Match *.

2023-05-06 ITSI-30026 Event generated from Provider are not getting grouped on Federated Search head

Workaround:
Event generated from provider gets grouped through the rule engine periodic backfill.
2023-03-30 ITSI-29292 Update information in Skipped Events panel in Event Analytics Monitoring Dashboard
2023-03-23 ITSI-29214 Episode Detail Panel does not display full name of user correctly
2023-03-14 ITSI-29095 Episode Detail Dashboard does not show updated token values
2023-02-08 ITSI-28707 Color for custom severity is not displayed correctly in Correlation Search Builder, Notable Event Aggregation Policy Editor and Episode Review page
2023-01-16 ITSI-28046 Alert action configuration UI not loaded in ITSI when the count of alert actions exceed 30

Workaround:
Keep the count of alert actions in the instance below 30
2023-01-12 ITSI-28015 The episode link in "Share Episode" does not get updated in right click menu
2022-12-20 ITSI-27751 Episode Review arbitrary search filter with AND & OR conditions fail to match events under certain scenarios

Workaround:
Avoid using brackets (), extra whitespaces, the operator !=, and double quotes "" in the search filter
2022-12-06 ITSI-27595 Actions Rules field names in UI not keeping the upper case upon save
2022-11-04 ITSI-27028 When Identifier Fields are specified for Notables and Smart Mode is enabled, the Episodes do not show the identifier fields
2022-10-25 ITSI-26825 Episode Review timeline search is triggered even when summary dashboard is closed which wastes resources.

Glass Table

Date filed Issue number Description
2023-04-11 ITSI-29450 variable should be pass between different widgets in glass table
2023-03-09 ITSI-29073 ITSI Glass Table tableFormat rowBackgroundColors - not coloring all the rows background
2023-01-10 ITSI-27969 Ad hoc search should work properly even if we add it after deleting the existing the kpi data source from the visualization

Workaround:
Remove the value of options field from glass table source code in visualization when you delete the KPI data source and add adhoc data source in same visualization.
2023-01-05 ITSI-27886 splunk.markdown adds unexpected background colour and text colour when leading spaces are used in text
2022-12-21 ITSI-27763 Assigning an ad-hoc search as a datasource to the empty viz, breaks the entire glass table page
2022-12-20 ITSI-27743 Drilldown and URL link in Glass Table may open double tabs/windows
2021-12-17 ITSI-20748 Service Swapping weirdness on Glass Table

KPI Base Searches

Date filed Issue number Description
2023-06-20 ITSI-31085 KPI Backfill searches run under 'Search' app context instead of ITSI/SA-ITOA app context
2022-10-05 ITSI-26497 app/itsi/kpi_base_searches_lister error

Workaround:
N/A

KPI Search Calculation

Date filed Issue number Description
2023-06-20 ITSI-31085 KPI Backfill searches run under 'Search' app context instead of ITSI/SA-ITOA app context
2023-02-24 ITSI-28886 mod_time and retirable appear as a metric_name in itsi_summary_metrics and unnecessarily creates extra datapoints
2022-12-16 ITSI-27721 KPI title surrounded with double quotes throws an error while running a KPI Generated Search

Maintenance Window

Date filed Issue number Description
2023-04-30 ITSI-29732 Maintenance Window page is stuck at Loading at Configurated Entities tab intermittently.
2023-03-12 ITSI-29078 Retired Entities not being filtered out of Maintenance Window List of Entities

Workaround:
N/A

Performance

Date filed Issue number Description
2023-04-26 ITSI-29672 KPI preview fails to render sometimes

Workaround:
NA
2023-03-16 ITSI-29110 The advanced filter on the Entity Management page times out when attempting to filter a large number of entities using the "does not match" filter for the entity type.

Workaround:

1. Reduce the number of results returned from the filter while using does not match filter on the entity type to avoid the high latency of the API. 2. Increased the timeout to avoid the 504 error when API is taking more than 5 minutes.

Role Based Access Controls

Date filed Issue number Description
2023-05-04 ITSI-30017 A user in itoa_user role cannot open ITSI homeview in SHC.

Workaround:
We have to add the list_search_head_clustering capability to the default authorize.conf.

Service Analyzer

Date filed Issue number Description
2023-06-07 ITSI-30580 When the dbconnect app is installed, non-admin ITSI users cannot access their homepage but are routed to the upgrade page.

Workaround:
Add the db_connect_read_app_conf capability to the custom user with a non-admin role. Enable this capability in the default authorize.conf file.
2023-02-17 ITSI-28826 Changes to health score color values in threshold_labels.conf do not appear in the service analyzer.
2022-10-07 ITSI-26544 Service Analyzer returns no data because join_kpi_info macro's sub search hits the 50K limit

Service Definition

Date filed Issue number Description
2023-04-12 ITSI-29486 Block users from saving a service with empty field in entity rules
2023-02-15 ITSI-28820, ITSI-28804 An ITSI service created from a service template with no defined filter condition will cause the service to add all entities.

Workaround:
*Disabled all the services that have an entity filter rule set to matchesblank to a blank value.
  • Have a valid filter rule defined in the services that have values in the *values* field.

Service Templates

Date filed Issue number Description
2023-05-16 ITSI-30132 Backport the bug fixes for 4.15.2
2023-03-21 ITSI-29200 Change in the threshold value of a KPI in a service template does not update in the services linked to it.

Workaround:
In the threshold editor, create a new temporary time policy, save, and select *Replace all KPI thresholds.* This will force the propagation of all time policies to all linked services. After this occurs, you can also delete the extra policy.
2023-02-15 ITSI-28820, ITSI-28804 An ITSI service created from a service template with no defined filter condition will cause the service to add all entities.

Workaround:
*Disabled all the services that have an entity filter rule set to matchesblank to a blank value.
  • Have a valid filter rule defined in the services that have values in the *values* field.
2022-10-18 ITSI-26757 Refresh queue is overriding base service template object while linking the service to it for more than 15 concurrent service creation.

Uncategorized issues

Date filed Issue number Description
2024-01-23 ITSI-34041 ITSI Episode view triggers a search to populate linked tickets, that is looking back to epoch time=1 second till now
2023-12-07 ITSI-33278 Cannot create a correlation search with all special character
2023-11-15 ITSI-33113 Bulk Acknowledge Episode can be executed for already Acknowledged episode by another user
2023-08-02 ITSI-31555, ITSI-31464 the ITSI integration create SNOW tickets with SPL instead of INC prefix when using Episode Action with custom endpoints with ServiceNow_TA version 7.6.0

Workaround:
Until bug in service now ADDON 7.6 bug (ADDON-64098 & ADDON-63502 ) are resolved, to avoid the issue, in ITSI, do not specify a custom endpoint in the action setup, keep the field empty.
2023-03-20 ITSI-29133 Episode Review dashboard panel for Noise reduction should not show "Missing property: majorValue"
2023-02-17 ITSI-28829 The timebased breaking event replaces the episode information fields.
2023-02-08 ITSI-28713 Removed AWS configurations from props.conf file
2023-01-12 ITSI-28026 "Show Alternative Views" UI toggle too small
2023-01-09 ITSI-27961 Bidirectional Ticketing Correlation Search hits "subsearch limit of 50000 reached" when the collection itsi_notable_event_ticketing has more than 50000 entries

Workaround:
# Navigate to ITSI -> Configuration -> Correlation Searches
  1. Click on Bidirectional Ticketing
  2. Paste the following search in the Search field and then click on Save. Also enable the CS if it has been disabled

{noformat}| datamodel Ticket_Management Incident search | rename All_Ticket_Management.ticket_id as ticket_id | join ticket_id [search sourcetype="snow:incident" index="<snow_index>" | where _indextime > now() - <max_lookback_time>] | lookup itsi_notable_event_external_ticket tickets.ticket_id as ticket_id OUTPUTNEW tickets.ticket_system event_id | where isnotnull(event_id) | rename tickets.* as * | eventstats values(event_id) as group_id last(ticket_system) as ticket_system by ticket_id | fields - dv_* | table * | makemv group_id | mvexpand group_id | eval bidirectional_ticketing=1, snow_hash = number + "!" + group_id + "!" + sys_updated_on | search NOT [| search index="itsi_tracked_alerts" | fields snow_hash] | dedup snow_hash{noformat}

Change the placeholders {{<snow_index>}} and {{<max_lookback_time>}} in the above search with values according to the customer's requirements

2023-01-06 ITSI-27928, ITSI-27925 Private Episodes should be created or read even if the capabilities are not provided
2023-01-02 ITSI-27863 Vital metrics show up as N/A for individual entities in Entity Overview Page

Workaround:
The macro has the id field which is getting used in all entity types and saved searches. Remove the id field from the vital metrics SPL and split by fields, or rename this field.
2022-12-26 ITSI-27798 Correlation Search Count is not visible in UI
2022-12-20 ITSI-27741 When closing episodes in bulk, episodes with different statuses display as closed but aren't actually closed.

Workaround:
During the bulk update of the episodes from the UI, make sure that all the Episodes selected for the bulk update at a time have same Status.
2022-12-08 ITSI-27627 Correlation search - count API throws "500 internal server error" when filter is performed on the name which doesn't match with any search
2022-12-08 ITSI-27623 Episode reviews page shows Access Denied for filters with objects no longer existed
2022-12-07 ITSI-27599 Status 414 when attempting to retire 12000 entities

Workaround:
Try to retire entities in batch of 7000 entities.
2022-12-06 ITSI-27586 EA Smart Recycling in retention policy not considering all end status in case of custom configurations
2022-12-05 ITSI-27575 Common Fields not displayed for Episode in Episode Review with Workload Management for all-time searches
2022-11-28 ITSI-27815 Invalid token for Auto Generated ITSI Event Management Token in Noah stacks

Workaround:
# Navigate to Settings → Data inputs → HTTP Event Collector
  1. For the input Template:Auto Generated ITSI Event Management Token, click on "Delete". This will delete the duplicate token value and within few seconds it will be replaced by the correct one
2022-11-22 ITSI-27450, ITSI-27449, ITSI-27451 is_partial_data=0 is not working as Documented for maintenance_services_interface/<object_type>/<_key>

Workaround:
Use Template:Is partial data= to use it as Template:Is partial data=0
2022-11-22 ITSI-27449, ITSI-27450 The is_partial_data=0 is not working as Documented for itoa_interface/<object_type>/<_key> POST call

Workaround:
Use Template:Is partial data= to use it as Template:Is partial data=0
2022-11-17 ITSI-27427 When episode is updated and auto-refresh is run, the left panel shows result which should not be the case
2022-10-11 ITSI-26585 Entities status is getting "Unstable" from "Active" when installing SA4CP 1.7.0 with ITEW

Workaround:
# Go to Settings → Searches, Reports, and Alerts
  1. Select App → Content Pack for ITSI Monitoring and Alerting (DA-ITSI-CP-monitoring-alerting)
  2. Select Owner → All
  3. Click on the saved search → ITSI Import Objects - itsi_entity_name_normalizer
  4. Edit the search to → | inputlookup itsi_entities | eval entity_name=title
  5. Save the saved search
2022-09-06 ITSI-26046 NumberFormatException causing Episodes to remain unbroken when NEAP is time-based and Episode Severity set to Same as Highest Severity

Workaround:
The customer will be able to manually close the episodes.

IMPORTANT: the outputlookup command is dangerous when used with the kvstore. It will overwrite the contents of the entire kvstore collection with the search results if the Template:Append=true flag is not set. The customer should make a backup before running the command.

Search to generate the objects to push to kvstore. Please run this search for the past 30 days.

{noformat}`itsi_event_management_group_index` | stats latest(owner) as owner, latest(severity) as severity, latest(status) as status, latest(itsi_instruction) as instruction by itsi_group_id | eval index_owner=owner, index_severity=severity, index_status=status, event_identifier_hash=itsi_group_id | fields index_owner, index_severity, index_status, itsi_group_id, instruction, event_identifier_hash | eval _key=itsi_group_id | lookup itsi_notable_group_system_lookup _key OUTPUT mod_time | lookup itsi_notable_group_user_lookup _key OUTPUT owner severity status | search NOT status=* AND mod_time=* | eval owner=index_owner, severity=index_severity, status=index_status, object_type="notable_group_user" | fields - index_owner, index_severity, index_status

{noformat}

If results look correct append the following Template:Outputlookup command and re-run search:

{noformat}| outputlookup itsi_notable_group_user_lookup append=true key_field=itsi_group_id{noformat}

This search should ideally update these Episodes:

"2a617192-1858-4219-aba8-ed7b777f3035" "ad3ec87e-05c2-4b1c-8ca9-c854ac6f6725" "ccfa9689-a4e8-460e-a001-45e6891361a8"

2022-07-11 ITSI-24902 ITSI entity management functionality flags previously detected entities as unstable after upgrade to newer versions.

Workaround:
Delete the existing entities and let the existing entities be re-discovered during next run of the discovery search.
2021-09-01 ITSI-18709 ITSI redirects to suite_redirect 500 Internal Server Error - because of python library isolation between apps

Workaround:
Step 1: Identify all the splunklib directories within the splunk apps directory using command find . -name 'splunklib' | xargs -r ls -lah.

Step 2: For each directory listed in step 1, check if file six.py is present.

Step 3: Copy the six.py from an existing splunklib directory into all the missing directories.

Step 4: Clean the cached files using find . -name "*.pyc" -delete

Step 5: Restart Splunk on the ITE Work or ITSI search head.

2021-08-25 ITSI-18574 Base search "head" command is removing relevant event data for snapshots, probably depending on the time you look at it
2021-08-22 ITSI-18480 ITSI license is checker unable to parse descriptions with multiple lines in server.conf

Workaround:
Check all the licenses and the license-pools in the license-manager, and find the one that contains custom Template:Description or notes. If they contains linebreaks, remove the linebreaks from the notes.
2019-05-30 ITSI-3322 If you add a correlation search in ITSI which contains a sub-search returning into an eval, you get a message "Invalid search string: This search cannot be parsed when parse_only is set to true."

Workaround:
You can't use a sub-search returning into an eval in a correlation search. As a workaround, create and save a basic correlation search with all of the information you want outside of the search. Then as an admin user, go to Settings > Searches, reports, and alerts and open the correlation search you just created. Add the sub-search you were trying to add there.

All ITSI Modules

Publication date Issue number Description
2017-03-21 ITOA-7585 When you bulk add services and an error caused by the racing condition occurs, the incorrect message "itsi_module does not exist" is displayed.
2017-03-07 MOD-979 KPIs do not have consistent backfill settings across all modules.
2017-01-17 MOD-452 The Analyze KPI button on the Service Details page is broken.
2017-01-17 MOD-402 The Export to PDF option does not work in the drilldown to a module.
2017-01-17 MOD-296 The extendable tab XML generator REST endpoint is located in DA-ITSI-OS instead of in common components where it can be used by all modules.
2017-01-17 MOD-591 ITSI displays a misleading error message when a KPI template contains a field that cannot be resolved.
2017-01-17 MOD-498 There is no upper limit to the number of characters a KPI title or description can contain. Long strings can negatively affect performance.
2017-01-17 MOD-309 The Gruntfile.js included in ITSI modules uses double quotes instead of single quotes, which does not conform to the standard for all JavaScript files.
2017-04-17 MOD-2002 When you drilldown from the Events tab, an "Invalid earliest_time" error occurs.


Workaround:
Disable drilldown from the Events tab.

2017-01-17 MOD-439 Some modules do not have descriptions for saved searches.

Application Server Module

Publication date Issue number Description
2017-01-27 MOD-492 If you reuse the same panel within a dashboard, the duplicate panel does not display any event data.

Cloud Services Module

There are no known issues for this release.

Database Module

Publication date Issue number Description
2017-01-17 MOD-586 When a lookup is not configured for TA-Microsoft-SqlServer, ITSI displays a misleading error message on the server drilldown page.

End User Experience Module

There are no known issues for this release.

Load Balancer Module

Publication date Issue number Description
2017-01-27 MOD-492 If you reuse the same panel within a dashboard, the duplicate panel does not display any event data.

Operating System Module

Publication date Issue number Description
2017-04-13 MOD-555 The Storage Free Space % base search runs every minute while the Linux df command runs every 5 minutes. This causes data gaps.
2017-04-10 MOD-1964 Windows data for memory free space is collected at different intervals than the Memory Free % KPI.
2017-01-17 MOD-1398 Line, stack, and area charts do not display a metric gap when no metrics are available during a time period.

Storage Module

There are no known issues for this release.

Virtualization Module

There are no known issues for this release.

Web Server Module

Publication date Issue number Description
2017-03-17 MOD-320 Some KPI ad hoc searches transform data with the stats command and do not retain time fields. The KPIs do not render anything and do not show thresholding details.
2017-03-17 MOD-538 When you add a new tab with panels and refresh the page, the page breaks.
Last modified on 26 February, 2024
PREVIOUS
Fixed issues in Splunk IT Service Intelligence
  NEXT
Removed features in Splunk IT Service Intelligence

This documentation applies to the following versions of Splunk® IT Service Intelligence: 4.15.0


Was this documentation topic helpful?


You must be logged into splunk.com in order to post comments. Log in now.

Please try to keep this discussion focused on the content covered in this documentation topic. If you have a more general question about Splunk functionality or are experiencing a difficulty with Splunk, consider posting a question to Splunkbase Answers.

0 out of 1000 Characters