Splunk® IT Service Intelligence

Release Notes

Acrobat logo Download manual as PDF


This documentation does not apply to the most recent version of Splunk® IT Service Intelligence. For documentation on the most recent version, go to the latest release.
Acrobat logo Download topic as PDF

Known issues in Splunk IT Service Intelligence

IT Service Intelligence (ITSI) version 4.14.0 has the following known issues and workarounds.

Highlighted issues

Date filed Issue number Description
2023-05-06 ITSI-30026 Event generated from Provider are not getting grouped on Federated Search head

Workaround:
Event generated from provider gets grouped through the rule engine periodic backfill.
2022-09-07 ITSI-26097 Entities and vital metrics are not populating on federated search setup

Workaround:
# Workaround for enabling entity discovery with federated search setup:

Change |makeresults to |makeresults | head 1 in saved searches from SPLUNK_HOME/etc/apps/itsi/(default) and (local)/savedsearch.conf.

Alternatively, you can make the change through the IT Service Intelligence interface by selecting *Settings > Searches, reports and alerts*, and then searching for the saved search name on the Searches, Reports, and Alerts page. For example, to discover *nix entities, you need go to '*ITSI Import Object - OS*' and revise '|*makeresults*' to become '|makeresults | head 1'

2. Workaround to ensure that vital metrics populate with federated search setup:

a. Change '|makeresults' to '|makeresults | head 1' in the following two macros 

i. gen_eval_fields(1)
ii. gen_as_fields(2)

Adaptive Thresholding

Date filed Issue number Description
2023-04-26 ITSI-29672 KPI preview fails to render sometimes

Workaround:
NA
2023-01-03 ITSI-27867 In Adaptive Thresholding Clicking on apply button shows any warning as errors in UI.
2022-08-23 ITSI-25903 Threshold Template Sync Fails with Empty Alert Values in threshold template

Backup/Restore and Migration Issues

Date filed Issue number Description
2023-02-28 ITSI-28926 kvstore_to_json.py restore operations do not remove existing services
2022-09-15 ITSI-26204 ITSI Default Scheduled Backup taking hours to complete after upgrade to 4.11.5 (it used to be minutes)

Workaround:
* Run the below curl command to delete the entry in the collection Template:Itsi migration status

{noformat}curl -ku admin https://localhost:8089/servicesNS/nobody/SA-ITOA/storage/collections/data/itsi_migration_status -X DELETE{noformat}

Entities

Date filed Issue number Description
2022-09-07 ITSI-26097 Entities and vital metrics are not populating on federated search setup

Workaround:
# Workaround for enabling entity discovery with federated search setup:

Change |makeresults to |makeresults | head 1 in saved searches from SPLUNK_HOME/etc/apps/itsi/(default) and (local)/savedsearch.conf.

Alternatively, you can make the change through the IT Service Intelligence interface by selecting *Settings > Searches, reports and alerts*, and then searching for the saved search name on the Searches, Reports, and Alerts page. For example, to discover *nix entities, you need go to '*ITSI Import Object - OS*' and revise '|*makeresults*' to become '|makeresults | head 1'

2. Workaround to ensure that vital metrics populate with federated search setup:

a. Change '|makeresults' to '|makeresults | head 1' in the following two macros 

i. gen_eval_fields(1)
ii. gen_as_fields(2)

Entity Rules

Date filed Issue number Description
2023-02-23 ITSI-28871 Entity filter rule considering empty value as a wildcard (*)

Notable Events

Date filed Issue number Description
2023-06-29 ITSI-31192 All Events tab does not render default columns if they are not present in NEAP JSON definition

Workaround:
# Use the latest ITSI Backup file to edit the NEAP JSON definition and remove the property Template:All events columns and restore the backup.
  1. Go to Episode Review page and add back all the desired columns
2023-06-19 ITSI-31057 host field value not visible to Rules Engine
2023-06-02 ITSI-30500 NEAP filtering criteria with value *(wildcard) does not satisfy the events which contain \n(line break) in the value

Workaround:
Add another negative filtering criteria for the field.

For example, if we have added a filtering criteria Source Matches * then event will not be picked up by a custom NEAP. You need to add another negative filtering criteria with the or condition so it will be similar to Source Does not Match *.

2023-05-06 ITSI-30026 Event generated from Provider are not getting grouped on Federated Search head

Workaround:
Event generated from provider gets grouped through the rule engine periodic backfill.
2023-02-08 ITSI-28707 Color for custom severity is not displayed correctly in Correlation Search Builder, Notable Event Aggregation Policy Editor and Episode Review page
2023-01-16 ITSI-28046 Alert action configuration UI not loaded in ITSI when the count of alert actions exceed 30

Workaround:
Keep the count of alert actions in the instance below 30
2023-01-12 ITSI-28015 The episode link in "Share Episode" does not get updated in right click menu
2022-12-20 ITSI-27751 Episode Review arbitrary search filter with AND & OR conditions fail to match events under certain scenarios

Workaround:
Avoid using brackets (), extra whitespaces, the operator !=, and double quotes "" in the search filter
2022-12-11 ITSI-27640 Event Analytics Monitoring dashboard does not list all NEAP

Workaround:
in Event Analytics Monitoring Dashboard → Aggregation Policy panel → Edit → edit the Aggregation policy search under Dynamic Options:

{noformat}| rest servicesNS/nobody/SA-ITOA/event_management_interface/notable_event_aggregation_policy splunk_server=* report_as=text 

 | spath input=value path={}.title | spath input=value path={}._key | rename {}.title as title 
 | rename {}._key as key | eval zipped=mvzip('key', 'title') | mvexpand zipped | eval zipped=split(zipped, ",") 
 | eval itsi_policy_id=mvindex(zipped,0), policy_title=mvindex(zipped, 1)

{noformat}

2022-11-04 ITSI-27028 When Identifier Fields are specified for Notables and Smart Mode is enabled, the Episodes do not show the identifier fields
2022-10-25 ITSI-26825 Episode Review timeline search is triggered even when summary dashboard is closed which wastes resources.

Notable Event Aggregation Policies

Date filed Issue number Description
2023-06-29 ITSI-31192 All Events tab does not render default columns if they are not present in NEAP JSON definition

Workaround:
# Use the latest ITSI Backup file to edit the NEAP JSON definition and remove the property Template:All events columns and restore the backup.
  1. Go to Episode Review page and add back all the desired columns
2023-06-19 ITSI-31057 host field value not visible to Rules Engine
2023-06-02 ITSI-30500 NEAP filtering criteria with value *(wildcard) does not satisfy the events which contain \n(line break) in the value

Workaround:
Add another negative filtering criteria for the field.

For example, if we have added a filtering criteria Source Matches * then event will not be picked up by a custom NEAP. You need to add another negative filtering criteria with the or condition so it will be similar to Source Does not Match *.

2023-05-06 ITSI-30026 Event generated from Provider are not getting grouped on Federated Search head

Workaround:
Event generated from provider gets grouped through the rule engine periodic backfill.
2023-02-08 ITSI-28707 Color for custom severity is not displayed correctly in Correlation Search Builder, Notable Event Aggregation Policy Editor and Episode Review page
2023-01-16 ITSI-28046 Alert action configuration UI not loaded in ITSI when the count of alert actions exceed 30

Workaround:
Keep the count of alert actions in the instance below 30
2023-01-12 ITSI-28015 The episode link in "Share Episode" does not get updated in right click menu
2022-12-20 ITSI-27751 Episode Review arbitrary search filter with AND & OR conditions fail to match events under certain scenarios

Workaround:
Avoid using brackets (), extra whitespaces, the operator !=, and double quotes "" in the search filter
2022-12-11 ITSI-27640 Event Analytics Monitoring dashboard does not list all NEAP

Workaround:
in Event Analytics Monitoring Dashboard → Aggregation Policy panel → Edit → edit the Aggregation policy search under Dynamic Options:

{noformat}| rest servicesNS/nobody/SA-ITOA/event_management_interface/notable_event_aggregation_policy splunk_server=* report_as=text 

 | spath input=value path={}.title | spath input=value path={}._key | rename {}.title as title 
 | rename {}._key as key | eval zipped=mvzip('key', 'title') | mvexpand zipped | eval zipped=split(zipped, ",") 
 | eval itsi_policy_id=mvindex(zipped,0), policy_title=mvindex(zipped, 1)

{noformat}

2022-11-04 ITSI-27028 When Identifier Fields are specified for Notables and Smart Mode is enabled, the Episodes do not show the identifier fields
2022-10-25 ITSI-26825 Episode Review timeline search is triggered even when summary dashboard is closed which wastes resources.

Glass Table

Date filed Issue number Description
2023-01-10 ITSI-27969 Ad hoc search should work properly even if we add it after deleting the existing the kpi data source from the visualization

Workaround:
Remove the value of options field from glass table source code in visualization when you delete the KPI data source and add adhoc data source in same visualization.
2023-01-05 ITSI-27886 splunk.markdown adds unexpected background colour and text colour when leading spaces are used in text
2022-12-20 ITSI-27743 Drilldown and URL link in Glass Table may open double tabs/windows
2022-07-29 ITSI-25262 Font size adjustments and drilldowns for text are not working properly for glass tables after upgrading to ITSI 4.13.1

Workaround:
Issue 1: The font size is not adjustable.

Font size can be adjusted in splunk.markdown at some level with use of the H button from the UI.

# Heading level 1 ## Heading level 2 ### Heading level 3

Issue 2: Drilldown is not supported.

A custom URL can be used in splunk.markdown in place of the drilldown.

2021-12-17 ITSI-20748 Service Swapping weirdness on Glass Table

KPI Base Searches

Date filed Issue number Description
2023-06-20 ITSI-31085 KPI Backfill searches run under 'Search' app context instead of ITSI/SA-ITOA app context
2022-10-05 ITSI-26497 app/itsi/kpi_base_searches_lister error

Workaround:
N/A
2022-08-23 ITSI-25903 Threshold Template Sync Fails with Empty Alert Values in threshold template
2022-08-16 ITSI-25834 Not able to create KPIs from Metric Based KPI Base Search

Workaround:
Need to create KPI with metric based search instead of metric based KPI Base Search.

KPI Search Calculation

Date filed Issue number Description
2023-06-20 ITSI-31085 KPI Backfill searches run under 'Search' app context instead of ITSI/SA-ITOA app context
2023-02-24 ITSI-28886 mod_time and retirable appear as a metric_name in itsi_summary_metrics and unnecessarily creates extra datapoints
2022-09-19 ITSI-26229 ITSI Service Analyzer Dashboard can't filter some services.

Workaround:
*Filter the service with a wildcard, for example, "PTTOR BSM E-Order DB*"
  • Search for the service in the Go To Service field.
2022-08-16 ITSI-25834 Not able to create KPIs from Metric Based KPI Base Search

Workaround:
Need to create KPI with metric based search instead of metric based KPI Base Search.

Performance

Date filed Issue number Description
2023-04-26 ITSI-29672 KPI preview fails to render sometimes

Workaround:
NA

Service Analyzer

Date filed Issue number Description
2022-10-07 ITSI-26544 Service Analyzer returns no data because join_kpi_info macro's sub search hits the 50K limit
2022-09-19 ITSI-26229 ITSI Service Analyzer Dashboard can't filter some services.

Workaround:
*Filter the service with a wildcard, for example, "PTTOR BSM E-Order DB*"
  • Search for the service in the Go To Service field.

Service Health Score

Date filed Issue number Description
2022-09-28 ITSI-26376 Large number of KPI caused the service_health_metrics_monitor sub search to hit the 50000 default limit, causing discrepancies in values in Service Health Score alert_level in itsi_summary_metrics versus itsi_summary indexes.

Workaround:
Increase the limits.conf to adjust to the total number of KPIs in the subsearch of service_health_metrics_monitor. See example for a customer with 50000-70000 KPI objects.

{{[join] }} Template:Subsearch maxout = 75000 {{#default was 50000 }} {{[searchresults] }} Template:Maxresultrows = 75000 {{ # default was 50000}}

Service Templates

Date filed Issue number Description
2022-08-23 ITSI-25903 Threshold Template Sync Fails with Empty Alert Values in threshold template

Uncategorized issues

Date filed Issue number Description
2023-02-17 ITSI-28829 The timebased breaking event replaces the episode information fields.
2023-01-12 ITSI-28026 "Show Alternative Views" UI toggle too small
2023-01-09 ITSI-27961 Bidirectional Ticketing Correlation Search hits "subsearch limit of 50000 reached" when the collection itsi_notable_event_ticketing has more than 50000 entries

Workaround:
# Navigate to ITSI -> Configuration -> Correlation Searches
  1. Click on Bidirectional Ticketing
  2. Paste the following search in the Search field and then click on Save. Also enable the CS if it has been disabled

{noformat}| datamodel Ticket_Management Incident search | rename All_Ticket_Management.ticket_id as ticket_id | join ticket_id [search sourcetype="snow:incident" index="<snow_index>" | where _indextime > now() - <max_lookback_time>] | lookup itsi_notable_event_external_ticket tickets.ticket_id as ticket_id OUTPUTNEW tickets.ticket_system event_id | where isnotnull(event_id) | rename tickets.* as * | eventstats values(event_id) as group_id last(ticket_system) as ticket_system by ticket_id | fields - dv_* | table * | makemv group_id | mvexpand group_id | eval bidirectional_ticketing=1, snow_hash = number + "!" + group_id + "!" + sys_updated_on | search NOT [| search index="itsi_tracked_alerts" | fields snow_hash] | dedup snow_hash{noformat}

Change the placeholders {{<snow_index>}} and {{<max_lookback_time>}} in the above search with values according to the customer's requirements

2023-01-06 ITSI-27928, ITSI-27925 Private Episodes should be created or read even if the capabilities are not provided
2022-12-08 ITSI-27627 Correlation search - count API throws "500 internal server error" when filter is performed on the name which doesn't match with any search
2022-10-03 ITSI-26442 Unable to download an authored content pack on a search head cluster.

Workaround:
Download the file from the search head captain. You can find the search head captain by running the following command: 

{{SPLUNK_SERVER}}/services/shcluster/captain/info?output_mode=json</code>

2022-09-13 ITSI-26147 High memory usage by backend processes cause ITSI to become unresponsive after upgrading to 4.14.0.
2022-09-06 ITSI-26046 NumberFormatException causing Episodes to remain unbroken when NEAP is time-based and Episode Severity set to Same as Highest Severity

Workaround:
The customer will be able to manually close the episodes.

IMPORTANT: the outputlookup command is dangerous when used with the kvstore. It will overwrite the contents of the entire kvstore collection with the search results if the Template:Append=true flag is not set. The customer should make a backup before running the command.

Search to generate the objects to push to kvstore. Please run this search for the past 30 days.

{noformat}`itsi_event_management_group_index` | stats latest(owner) as owner, latest(severity) as severity, latest(status) as status, latest(itsi_instruction) as instruction by itsi_group_id | eval index_owner=owner, index_severity=severity, index_status=status, event_identifier_hash=itsi_group_id | fields index_owner, index_severity, index_status, itsi_group_id, instruction, event_identifier_hash | eval _key=itsi_group_id | lookup itsi_notable_group_system_lookup _key OUTPUT mod_time | lookup itsi_notable_group_user_lookup _key OUTPUT owner severity status | search NOT status=* AND mod_time=* | eval owner=index_owner, severity=index_severity, status=index_status, object_type="notable_group_user" | fields - index_owner, index_severity, index_status

{noformat}

If results look correct append the following Template:Outputlookup command and re-run search:

{noformat}| outputlookup itsi_notable_group_user_lookup append=true key_field=itsi_group_id{noformat}

This search should ideally update these Episodes:

"2a617192-1858-4219-aba8-ed7b777f3035" "ad3ec87e-05c2-4b1c-8ca9-c854ac6f6725" "ccfa9689-a4e8-460e-a001-45e6891361a8"

2022-07-12 ITSI-24964 ITSI Searches ("Date Range", "Date & Time Range") do not honor auto-generated values; new Real-time search option fails (tstats not supported in a real-time search)

Workaround:
For Date & Time selections: manually enter/replace any portion of the auto-filled date for both start and end dates (even if replacing with the same value); or, select date from the calendar dropdown.

For Date selection only: No workaround found so far.

2021-10-25 ITSI-19489 The Next Scheduled Time for entity management policies is based on the system time zone, instead of the user's current time zone.
2021-09-01 ITSI-18709 ITSI redirects to suite_redirect 500 Internal Server Error - because of python library isolation between apps

Workaround:
Step 1: Identify all the splunklib directories within the splunk apps directory using command find . -name 'splunklib' | xargs -r ls -lah.

Step 2: For each directory listed in step 1, check if file six.py is present.

Step 3: Copy the six.py from an existing splunklib directory into all the missing directories.

Step 4: Clean the cached files using find . -name "*.pyc" -delete

Step 5: Restart Splunk on the ITE Work or ITSI search head.

2021-08-22 ITSI-18480 ITSI license is checker unable to parse descriptions with multiple lines in server.conf

Workaround:
Check all the licenses and the license-pools in the license-manager, and find the one that contains custom Template:Description or notes. If they contains linebreaks, remove the linebreaks from the notes.
2019-05-30 ITSI-3322 If you add a correlation search in ITSI which contains a sub-search returning into an eval, you get a message "Invalid search string: This search cannot be parsed when parse_only is set to true."

Workaround:
You can't use a sub-search returning into an eval in a correlation search. As a workaround, create and save a basic correlation search with all of the information you want outside of the search. Then as an admin user, go to Settings > Searches, reports, and alerts and open the correlation search you just created. Add the sub-search you were trying to add there.
Last modified on 12 January, 2024
PREVIOUS
Fixed issues in Splunk IT Service Intelligence 		  NEXT
Removed features in Splunk IT Service Intelligence

This documentation applies to the following versions of Splunk® IT Service Intelligence: 4.14.0 Cloud only

Was this documentation topic helpful?


You must be logged into splunk.com in order to post comments. Log in now.

Please try to keep this discussion focused on the content covered in this documentation topic. If you have a more general question about Splunk functionality or are experiencing a difficulty with Splunk, consider posting a question to Splunkbase Answers.

0 out of 1000 Characters