Known issues in Splunk IT Service Intelligence
IT Service Intelligence (ITSI) version 4.14.0 has the following known issues and workarounds.
Highlighted issues
Date filed | Issue number | Description |
---|---|---|
2023-05-06 | ITSI-30026 | Event generated from Provider are not getting grouped on Federated Search head Workaround: Event generated from provider gets grouped through the rule engine periodic backfill. |
2022-09-07 | ITSI-26097 | Entities and vital metrics are not populating on federated search setup Workaround: # Workaround for enabling entity discovery with federated search setup: Change Alternatively, you can make the change through the IT Service Intelligence interface by selecting *Settings > Searches, reports and alerts*, and then searching for the saved search name on the Searches, Reports, and Alerts page. For example, to discover *nix entities, you need go to '*ITSI Import Object - OS*' and revise '|*makeresults*' to become '|makeresults | head 1' 2. Workaround to ensure that vital metrics populate with federated search setup: a. Change '|makeresults' to '|makeresults | head 1' in the following two macros i. |
Adaptive Thresholding
Date filed | Issue number | Description |
---|---|---|
2023-04-26 | ITSI-29672 | KPI preview fails to render sometimes Workaround: NA |
2023-01-03 | ITSI-27867 | In Adaptive Thresholding Clicking on apply button shows any warning as errors in UI. |
2022-08-23 | ITSI-25903 | Threshold Template Sync Fails with Empty Alert Values in threshold template |
Backup/Restore and Migration Issues
Date filed | Issue number | Description |
---|---|---|
2023-02-28 | ITSI-28926 | kvstore_to_json.py restore operations do not remove existing services |
2022-09-15 | ITSI-26204 | ITSI Default Scheduled Backup taking hours to complete after upgrade to 4.11.5 (it used to be minutes) Workaround: * Run the below curl command to delete the entry in the collection Template:Itsi migration status {noformat}curl -ku admin https://localhost:8089/servicesNS/nobody/SA-ITOA/storage/collections/data/itsi_migration_status -X DELETE{noformat} |
Entities
Date filed | Issue number | Description |
---|---|---|
2022-09-07 | ITSI-26097 | Entities and vital metrics are not populating on federated search setup Workaround: # Workaround for enabling entity discovery with federated search setup: Change Alternatively, you can make the change through the IT Service Intelligence interface by selecting *Settings > Searches, reports and alerts*, and then searching for the saved search name on the Searches, Reports, and Alerts page. For example, to discover *nix entities, you need go to '*ITSI Import Object - OS*' and revise '|*makeresults*' to become '|makeresults | head 1' 2. Workaround to ensure that vital metrics populate with federated search setup: a. Change '|makeresults' to '|makeresults | head 1' in the following two macros i. |
Entity Rules
Date filed | Issue number | Description |
---|---|---|
2023-02-23 | ITSI-28871 | Entity filter rule considering empty value as a wildcard (*) |
Notable Events
Date filed | Issue number | Description |
---|---|---|
2023-06-29 | ITSI-31192 | All Events tab does not render default columns if they are not present in NEAP JSON definition Workaround: # Use the latest ITSI Backup file to edit the NEAP JSON definition and remove the property Template:All events columns and restore the backup.
|
2023-06-19 | ITSI-31057 | host field value not visible to Rules Engine |
2023-06-02 | ITSI-30500 | NEAP filtering criteria with value *(wildcard) does not satisfy the events which contain \n(line break) in the value Workaround: Add another negative filtering criteria for the field. For example, if we have added a filtering criteria |
2023-05-06 | ITSI-30026 | Event generated from Provider are not getting grouped on Federated Search head Workaround: Event generated from provider gets grouped through the rule engine periodic backfill. |
2023-02-08 | ITSI-28707 | Color for custom severity is not displayed correctly in Correlation Search Builder, Notable Event Aggregation Policy Editor and Episode Review page |
2023-01-16 | ITSI-28046 | Alert action configuration UI not loaded in ITSI when the count of alert actions exceed 30 Workaround: Keep the count of alert actions in the instance below 30 |
2023-01-12 | ITSI-28015 | The episode link in "Share Episode" does not get updated in right click menu |
2022-12-20 | ITSI-27751 | Episode Review arbitrary search filter with AND & OR conditions fail to match events under certain scenarios Workaround: Avoid using brackets () , extra whitespaces, the operator != , and double quotes "" in the search filter |
2022-12-11 | ITSI-27640 | Event Analytics Monitoring dashboard does not list all NEAP Workaround: in Event Analytics Monitoring Dashboard → Aggregation Policy panel → Edit → edit the Aggregation policy search under Dynamic Options: {noformat}| rest servicesNS/nobody/SA-ITOA/event_management_interface/notable_event_aggregation_policy splunk_server=* report_as=text | spath input=value path={}.title | spath input=value path={}._key | rename {}.title as title | rename {}._key as key | eval zipped=mvzip('key', 'title') | mvexpand zipped | eval zipped=split(zipped, ",") | eval itsi_policy_id=mvindex(zipped,0), policy_title=mvindex(zipped, 1) {noformat} |
2022-11-04 | ITSI-27028 | When Identifier Fields are specified for Notables and Smart Mode is enabled, the Episodes do not show the identifier fields |
2022-10-25 | ITSI-26825 | Episode Review timeline search is triggered even when summary dashboard is closed which wastes resources. |
Notable Event Aggregation Policies
Date filed | Issue number | Description |
---|---|---|
2023-06-29 | ITSI-31192 | All Events tab does not render default columns if they are not present in NEAP JSON definition Workaround: # Use the latest ITSI Backup file to edit the NEAP JSON definition and remove the property Template:All events columns and restore the backup.
|
2023-06-19 | ITSI-31057 | host field value not visible to Rules Engine |
2023-06-02 | ITSI-30500 | NEAP filtering criteria with value *(wildcard) does not satisfy the events which contain \n(line break) in the value Workaround: Add another negative filtering criteria for the field. For example, if we have added a filtering criteria |
2023-05-06 | ITSI-30026 | Event generated from Provider are not getting grouped on Federated Search head Workaround: Event generated from provider gets grouped through the rule engine periodic backfill. |
2023-02-08 | ITSI-28707 | Color for custom severity is not displayed correctly in Correlation Search Builder, Notable Event Aggregation Policy Editor and Episode Review page |
2023-01-16 | ITSI-28046 | Alert action configuration UI not loaded in ITSI when the count of alert actions exceed 30 Workaround: Keep the count of alert actions in the instance below 30 |
2023-01-12 | ITSI-28015 | The episode link in "Share Episode" does not get updated in right click menu |
2022-12-20 | ITSI-27751 | Episode Review arbitrary search filter with AND & OR conditions fail to match events under certain scenarios Workaround: Avoid using brackets () , extra whitespaces, the operator != , and double quotes "" in the search filter |
2022-12-11 | ITSI-27640 | Event Analytics Monitoring dashboard does not list all NEAP Workaround: in Event Analytics Monitoring Dashboard → Aggregation Policy panel → Edit → edit the Aggregation policy search under Dynamic Options: {noformat}| rest servicesNS/nobody/SA-ITOA/event_management_interface/notable_event_aggregation_policy splunk_server=* report_as=text | spath input=value path={}.title | spath input=value path={}._key | rename {}.title as title | rename {}._key as key | eval zipped=mvzip('key', 'title') | mvexpand zipped | eval zipped=split(zipped, ",") | eval itsi_policy_id=mvindex(zipped,0), policy_title=mvindex(zipped, 1) {noformat} |
2022-11-04 | ITSI-27028 | When Identifier Fields are specified for Notables and Smart Mode is enabled, the Episodes do not show the identifier fields |
2022-10-25 | ITSI-26825 | Episode Review timeline search is triggered even when summary dashboard is closed which wastes resources. |
Glass Table
Date filed | Issue number | Description |
---|---|---|
2023-01-10 | ITSI-27969 | Ad hoc search should work properly even if we add it after deleting the existing the kpi data source from the visualization Workaround: Remove the value of options field from glass table source code in visualization when you delete the KPI data source and add adhoc data source in same visualization. |
2023-01-05 | ITSI-27886 | splunk.markdown adds unexpected background colour and text colour when leading spaces are used in text |
2022-12-20 | ITSI-27743 | Drilldown and URL link in Glass Table may open double tabs/windows |
2022-07-29 | ITSI-25262 | Font size adjustments and drilldowns for text are not working properly for glass tables after upgrading to ITSI 4.13.1 Workaround: Issue 1: The font size is not adjustable. Font size can be adjusted in splunk.markdown at some level with use of the H button from the UI.
Issue 2: Drilldown is not supported. A custom URL can be used in splunk.markdown in place of the drilldown. |
2021-12-17 | ITSI-20748 | Service Swapping weirdness on Glass Table |
KPI Base Searches
Date filed | Issue number | Description |
---|---|---|
2023-06-20 | ITSI-31085 | KPI Backfill searches run under 'Search' app context instead of ITSI/SA-ITOA app context |
2022-10-05 | ITSI-26497 | app/itsi/kpi_base_searches_lister error Workaround: N/A |
2022-08-23 | ITSI-25903 | Threshold Template Sync Fails with Empty Alert Values in threshold template |
2022-08-16 | ITSI-25834 | Not able to create KPIs from Metric Based KPI Base Search Workaround: Need to create KPI with metric based search instead of metric based KPI Base Search. |
KPI Search Calculation
Date filed | Issue number | Description |
---|---|---|
2023-06-20 | ITSI-31085 | KPI Backfill searches run under 'Search' app context instead of ITSI/SA-ITOA app context |
2023-02-24 | ITSI-28886 | mod_time and retirable appear as a metric_name in itsi_summary_metrics and unnecessarily creates extra datapoints |
2022-09-19 | ITSI-26229 | ITSI Service Analyzer Dashboard can't filter some services. Workaround: *Filter the service with a wildcard, for example, "PTTOR BSM E-Order DB*"
|
2022-08-16 | ITSI-25834 | Not able to create KPIs from Metric Based KPI Base Search Workaround: Need to create KPI with metric based search instead of metric based KPI Base Search. |
Performance
Date filed | Issue number | Description |
---|---|---|
2023-04-26 | ITSI-29672 | KPI preview fails to render sometimes Workaround: NA |
Service Analyzer
Date filed | Issue number | Description |
---|---|---|
2022-10-07 | ITSI-26544 | Service Analyzer returns no data because join_kpi_info macro's sub search hits the 50K limit |
2022-09-19 | ITSI-26229 | ITSI Service Analyzer Dashboard can't filter some services. Workaround: *Filter the service with a wildcard, for example, "PTTOR BSM E-Order DB*"
|
Service Health Score
Date filed | Issue number | Description |
---|---|---|
2022-09-28 | ITSI-26376 | Large number of KPI caused the service_health_metrics_monitor sub search to hit the 50000 default limit, causing discrepancies in values in Service Health Score alert_level in itsi_summary_metrics versus itsi_summary indexes. Workaround: Increase the limits.conf to adjust to the total number of KPIs in the subsearch of service_health_metrics_monitor. See example for a customer with 50000-70000 KPI objects. {{[join] }}
Template:Subsearch maxout = 75000
{{#default was 50000 }}
{{[searchresults] }}
Template:Maxresultrows = 75000
{{ # default was 50000}} |
Service Templates
Date filed | Issue number | Description |
---|---|---|
2022-08-23 | ITSI-25903 | Threshold Template Sync Fails with Empty Alert Values in threshold template |
Uncategorized issues
Date filed | Issue number | Description |
---|---|---|
2023-02-17 | ITSI-28829 | The timebased breaking event replaces the episode information fields. |
2023-01-12 | ITSI-28026 | "Show Alternative Views" UI toggle too small |
2023-01-09 | ITSI-27961 | Bidirectional Ticketing Correlation Search hits "subsearch limit of 50000 reached" when the collection itsi_notable_event_ticketing has more than 50000 entries Workaround: # Navigate to ITSI -> Configuration -> Correlation Searches
{noformat}| datamodel Ticket_Management Incident search | rename All_Ticket_Management.ticket_id as ticket_id | join ticket_id [search sourcetype="snow:incident" index="<snow_index>" | where _indextime > now() - <max_lookback_time>] | lookup itsi_notable_event_external_ticket tickets.ticket_id as ticket_id OUTPUTNEW tickets.ticket_system event_id | where isnotnull(event_id) | rename tickets.* as * | eventstats values(event_id) as group_id last(ticket_system) as ticket_system by ticket_id | fields - dv_* | table * | makemv group_id | mvexpand group_id | eval bidirectional_ticketing=1, snow_hash = number + "!" + group_id + "!" + sys_updated_on | search NOT [| search index="itsi_tracked_alerts" | fields snow_hash] | dedup snow_hash{noformat} Change the placeholders {{<snow_index>}} and {{<max_lookback_time>}} in the above search with values according to the customer's requirements |
2023-01-06 | ITSI-27928, ITSI-27925 | Private Episodes should be created or read even if the capabilities are not provided |
2022-12-08 | ITSI-27627 | Correlation search - count API throws "500 internal server error" when filter is performed on the name which doesn't match with any search |
2022-10-03 | ITSI-26442 | Unable to download an authored content pack on a search head cluster. Workaround: Download the file from the search head captain. You can find the search head captain by running the following command: {{SPLUNK_SERVER}}/services/shcluster/captain/info?output_mode=json</code> |
2022-09-13 | ITSI-26147 | High memory usage by backend processes cause ITSI to become unresponsive after upgrading to 4.14.0. |
2022-09-06 | ITSI-26046 | NumberFormatException causing Episodes to remain unbroken when NEAP is time-based and Episode Severity set to Same as Highest Severity Workaround: The customer will be able to manually close the episodes. IMPORTANT: the outputlookup command is dangerous when used with the kvstore. It will overwrite the contents of the entire kvstore collection with the search results if the Template:Append=true flag is not set. The customer should make a backup before running the command. Search to generate the objects to push to kvstore. Please run this search for the past 30 days. {noformat}`itsi_event_management_group_index` | stats latest(owner) as owner, latest(severity) as severity, latest(status) as status, latest(itsi_instruction) as instruction by itsi_group_id | eval index_owner=owner, index_severity=severity, index_status=status, event_identifier_hash=itsi_group_id | fields index_owner, index_severity, index_status, itsi_group_id, instruction, event_identifier_hash | eval _key=itsi_group_id | lookup itsi_notable_group_system_lookup _key OUTPUT mod_time | lookup itsi_notable_group_user_lookup _key OUTPUT owner severity status | search NOT status=* AND mod_time=* | eval owner=index_owner, severity=index_severity, status=index_status, object_type="notable_group_user" | fields - index_owner, index_severity, index_status {noformat} If results look correct append the following Template:Outputlookup command and re-run search: {noformat}| outputlookup itsi_notable_group_user_lookup append=true key_field=itsi_group_id{noformat} This search should ideally update these Episodes: "2a617192-1858-4219-aba8-ed7b777f3035"
"ad3ec87e-05c2-4b1c-8ca9-c854ac6f6725"
"ccfa9689-a4e8-460e-a001-45e6891361a8" |
2022-07-12 | ITSI-24964 | ITSI Searches ("Date Range", "Date & Time Range") do not honor auto-generated values; new Real-time search option fails (tstats not supported in a real-time search) Workaround: For Date & Time selections: manually enter/replace any portion of the auto-filled date for both start and end dates (even if replacing with the same value); or, select date from the calendar dropdown. For Date selection only: No workaround found so far. |
2021-10-25 | ITSI-19489 | The Next Scheduled Time for entity management policies is based on the system time zone, instead of the user's current time zone. |
2021-09-01 | ITSI-18709 | ITSI redirects to suite_redirect 500 Internal Server Error - because of python library isolation between apps Workaround: Step 1: Identify all the splunklib directories within the splunk apps directory using command find . -name 'splunklib' | xargs -r ls -lah .
Step 2: For each directory listed in step 1, check if file Step 3: Copy the Step 4: Clean the cached files using Step 5: Restart Splunk on the ITE Work or ITSI search head. |
2021-08-22 | ITSI-18480 | ITSI license is checker unable to parse descriptions with multiple lines in server.conf Workaround: Check all the licenses and the license-pools in the license-manager, and find the one that contains custom Template:Description or notes. If they contains linebreaks, remove the linebreaks from the notes. |
2019-05-30 | ITSI-3322 | If you add a correlation search in ITSI which contains a sub-search returning into an eval, you get a message "Invalid search string: This search cannot be parsed when parse_only is set to true." Workaround: You can't use a sub-search returning into an eval in a correlation search. As a workaround, create and save a basic correlation search with all of the information you want outside of the search. Then as an admin user, go to Settings > Searches, reports, and alerts and open the correlation search you just created. Add the sub-search you were trying to add there. |
Fixed issues in Splunk IT Service Intelligence | Removed features in Splunk IT Service Intelligence |
This documentation applies to the following versions of Splunk® IT Service Intelligence: 4.14.0 Cloud only
Feedback submitted, thanks!