Known issues in Splunk IT Service Intelligence
IT Service Intelligence (ITSI) version 4.13.0 has the following known issues and workarounds.
Adaptive Thresholding
Date filed | Issue number | Description |
---|---|---|
2023-04-26 | ITSI-29672 | KPI preview fails to render sometimes Workaround: NA |
2023-01-03 | ITSI-27867 | In Adaptive Thresholding Clicking on apply button shows any warning as errors in UI. |
2022-08-23 | ITSI-25903 | Threshold Template Sync Fails with Empty Alert Values in threshold template |
Backup/Restore and Migration Issues
Date filed | Issue number | Description |
---|---|---|
2022-09-15 | ITSI-26204 | ITSI Default Scheduled Backup taking hours to complete after upgrade to 4.11.5 (it used to be minutes) Workaround: * Run the below curl command to delete the entry in the collection Template:Itsi migration status {noformat}curl -ku admin https://localhost:8089/servicesNS/nobody/SA-ITOA/storage/collections/data/itsi_migration_status -X DELETE{noformat} |
Notable Events
Date filed | Issue number | Description |
---|---|---|
2023-06-29 | ITSI-31192 | All Events tab does not render default columns if they are not present in NEAP JSON definition Workaround: # Use the latest ITSI Backup file to edit the NEAP JSON definition and remove the property Template:All events columns and restore the backup.
|
2023-02-08 | ITSI-28707 | Color for custom severity is not displayed correctly in Correlation Search Builder, Notable Event Aggregation Policy Editor and Episode Review page |
2023-01-16 | ITSI-28046 | Alert action configuration UI not loaded in ITSI when the count of alert actions exceed 30 Workaround: Keep the count of alert actions in the instance below 30 |
2023-01-12 | ITSI-28015 | The episode link in "Share Episode" does not get updated in right click menu |
2022-12-11 | ITSI-27640 | Event Analytics Monitoring dashboard does not list all NEAP Workaround: in Event Analytics Monitoring Dashboard → Aggregation Policy panel → Edit → edit the Aggregation policy search under Dynamic Options: {noformat}| rest servicesNS/nobody/SA-ITOA/event_management_interface/notable_event_aggregation_policy splunk_server=* report_as=text | spath input=value path={}.title | spath input=value path={}._key | rename {}.title as title | rename {}._key as key | eval zipped=mvzip('key', 'title') | mvexpand zipped | eval zipped=split(zipped, ",") | eval itsi_policy_id=mvindex(zipped,0), policy_title=mvindex(zipped, 1) {noformat} |
2022-12-06 | ITSI-27595 | Actions Rules field names in UI not keeping the upper case upon save |
2022-06-07 | ITSI-24488 | Rules engine search fails to start after upgrade to ITSI 4.13.0 Workaround: Move the jackson-core-2.10.0.jar and jackson-annotations-2.10.0.jar to the .bkup folder under $SPLUNK_HOME/etc/apps/SA-ITOA/lib/java/event_management/libs directory. |
2022-05-21 | ITSI-24325 | Error Message states, "One or more peers are down. Indexer cluster status may not be healthy" since ITSI upgrade |
Notable Event Aggregation Policies
Date filed | Issue number | Description |
---|---|---|
2023-06-29 | ITSI-31192 | All Events tab does not render default columns if they are not present in NEAP JSON definition Workaround: # Use the latest ITSI Backup file to edit the NEAP JSON definition and remove the property Template:All events columns and restore the backup.
|
2023-02-08 | ITSI-28707 | Color for custom severity is not displayed correctly in Correlation Search Builder, Notable Event Aggregation Policy Editor and Episode Review page |
2023-01-16 | ITSI-28046 | Alert action configuration UI not loaded in ITSI when the count of alert actions exceed 30 Workaround: Keep the count of alert actions in the instance below 30 |
2023-01-12 | ITSI-28015 | The episode link in "Share Episode" does not get updated in right click menu |
2022-12-11 | ITSI-27640 | Event Analytics Monitoring dashboard does not list all NEAP Workaround: in Event Analytics Monitoring Dashboard → Aggregation Policy panel → Edit → edit the Aggregation policy search under Dynamic Options: {noformat}| rest servicesNS/nobody/SA-ITOA/event_management_interface/notable_event_aggregation_policy splunk_server=* report_as=text | spath input=value path={}.title | spath input=value path={}._key | rename {}.title as title | rename {}._key as key | eval zipped=mvzip('key', 'title') | mvexpand zipped | eval zipped=split(zipped, ",") | eval itsi_policy_id=mvindex(zipped,0), policy_title=mvindex(zipped, 1) {noformat} |
2022-12-06 | ITSI-27595 | Actions Rules field names in UI not keeping the upper case upon save |
2022-06-07 | ITSI-24488 | Rules engine search fails to start after upgrade to ITSI 4.13.0 Workaround: Move the jackson-core-2.10.0.jar and jackson-annotations-2.10.0.jar to the .bkup folder under $SPLUNK_HOME/etc/apps/SA-ITOA/lib/java/event_management/libs directory. |
2022-05-21 | ITSI-24325 | Error Message states, "One or more peers are down. Indexer cluster status may not be healthy" since ITSI upgrade |
Glass Table
Date filed | Issue number | Description |
---|---|---|
2023-09-11 | ITSI-32016 | upgade from ITSI 4.11.5 -> 4.13.0 trigger errors on glasstables |
2023-01-10 | ITSI-27969 | Ad hoc search should work properly even if we add it after deleting the existing the kpi data source from the visualization Workaround: Remove the value of options field from glass table source code in visualization when you delete the KPI data source and add adhoc data source in same visualization. |
2023-01-05 | ITSI-27886 | splunk.markdown adds unexpected background colour and text colour when leading spaces are used in text |
2022-07-29 | ITSI-25262 | Font size adjustments and drilldowns for text are not working properly for glass tables after upgrading to ITSI 4.13.1 Workaround: Issue 1: The font size is not adjustable. Font size can be adjusted in splunk.markdown at some level with use of the H button from the UI.
Issue 2: Drilldown is not supported. A custom URL can be used in splunk.markdown in place of the drilldown. |
2021-12-17 | ITSI-20748 | Service Swapping weirdness on Glass Table |
KPI Base Searches
Date filed | Issue number | Description |
---|---|---|
2022-10-05 | ITSI-26497 | app/itsi/kpi_base_searches_lister error Workaround: N/A |
2022-08-23 | ITSI-25903 | Threshold Template Sync Fails with Empty Alert Values in threshold template |
2022-07-18 | ITSI-25037 | 'Add Metric' option not working for metric type search in KPI Base search creation |
2022-05-24 | ITSI-24346 | KPI Reports Incorrect Values due to Auto-Generated Entity Filtering Logic Workaround: Removing the "extra" search_type= comparator before executing the KPI search again with the (edited) auto-generated entity filter returns expected results. Without doing so, not all expected fields and rows are always present |
KPI Search Calculation
Date filed | Issue number | Description |
---|---|---|
2023-02-24 | ITSI-28886 | mod_time and retirable appear as a metric_name in itsi_summary_metrics and unnecessarily creates extra datapoints |
2022-05-24 | ITSI-24346 | KPI Reports Incorrect Values due to Auto-Generated Entity Filtering Logic Workaround: Removing the "extra" search_type= comparator before executing the KPI search again with the (edited) auto-generated entity filter returns expected results. Without doing so, not all expected fields and rows are always present |
2022-04-28 | ITSI-23284 | Deleted KPI lanes still showing in deep dive when the URL is refreshed. |
2022-04-21 | ITSI-23110 | When summary index has huge data KPI edit workflow takes a long time from Step-1 to Step-2. |
Maintenance Window
Date filed | Issue number | Description |
---|---|---|
2022-02-17 | ITSI-22148 | Unauthorized user can access maintenance window detail view for an existing maintenance window |
Performance
Date filed | Issue number | Description |
---|---|---|
2023-04-26 | ITSI-29672 | KPI preview fails to render sometimes Workaround: NA |
Role Based Access Controls
Date filed | Issue number | Description |
---|---|---|
2022-07-13 | ITSI-24979 | In Alerts and Episodes, users can view and access all saved episode review pages in 'Show Alternate Views' collapsible panel and can also delete any view of other users that is private |
Service Analyzer
Date filed | Issue number | Description |
---|---|---|
2023-02-17 | ITSI-28826 | Changes to health score color values in threshold_labels.conf do not appear in the service analyzer. |
2022-10-07 | ITSI-26544 | Service Analyzer returns no data because join_kpi_info macro's sub search hits the 50K limit |
Service Health Score
Date filed | Issue number | Description |
---|---|---|
2022-09-28 | ITSI-26376 | Large number of KPI caused the service_health_metrics_monitor sub search to hit the 50000 default limit, causing discrepancies in values in Service Health Score alert_level in itsi_summary_metrics versus itsi_summary indexes. Workaround: Increase the limits.conf to adjust to the total number of KPIs in the subsearch of service_health_metrics_monitor. See example for a customer with 50000-70000 KPI objects. {{[join] }}
Template:Subsearch maxout = 75000
{{#default was 50000 }}
{{[searchresults] }}
Template:Maxresultrows = 75000
{{ # default was 50000}} |
Service Templates
Date filed | Issue number | Description |
---|---|---|
2022-10-18 | ITSI-26757 | Refresh queue is overriding base service template object while linking the service to it for more than 15 concurrent service creation. |
2022-08-23 | ITSI-25903 | Threshold Template Sync Fails with Empty Alert Values in threshold template |
Uncategorized issues
Date filed | Issue number | Description |
---|---|---|
2023-01-09 | ITSI-27961 | Bidirectional Ticketing Correlation Search hits "subsearch limit of 50000 reached" when the collection itsi_notable_event_ticketing has more than 50000 entries Workaround: # Navigate to ITSI -> Configuration -> Correlation Searches
{noformat}| datamodel Ticket_Management Incident search | rename All_Ticket_Management.ticket_id as ticket_id | join ticket_id [search sourcetype="snow:incident" index="<snow_index>" | where _indextime > now() - <max_lookback_time>] | lookup itsi_notable_event_external_ticket tickets.ticket_id as ticket_id OUTPUTNEW tickets.ticket_system event_id | where isnotnull(event_id) | rename tickets.* as * | eventstats values(event_id) as group_id last(ticket_system) as ticket_system by ticket_id | fields - dv_* | table * | makemv group_id | mvexpand group_id | eval bidirectional_ticketing=1, snow_hash = number + "!" + group_id + "!" + sys_updated_on | search NOT [| search index="itsi_tracked_alerts" | fields snow_hash] | dedup snow_hash{noformat} Change the placeholders {{<snow_index>}} and {{<max_lookback_time>}} in the above search with values according to the customer's requirements |
2022-12-06 | ITSI-27586 | EA Smart Recycling in retention policy not considering all end status in case of custom configurations |
2022-09-06 | ITSI-26046 | NumberFormatException causing Episodes to remain unbroken when NEAP is time-based and Episode Severity set to Same as Highest Severity Workaround: The customer will be able to manually close the episodes. IMPORTANT: the outputlookup command is dangerous when used with the kvstore. It will overwrite the contents of the entire kvstore collection with the search results if the Template:Append=true flag is not set. The customer should make a backup before running the command. Search to generate the objects to push to kvstore. Please run this search for the past 30 days. {noformat}`itsi_event_management_group_index` | stats latest(owner) as owner, latest(severity) as severity, latest(status) as status, latest(itsi_instruction) as instruction by itsi_group_id | eval index_owner=owner, index_severity=severity, index_status=status, event_identifier_hash=itsi_group_id | fields index_owner, index_severity, index_status, itsi_group_id, instruction, event_identifier_hash | eval _key=itsi_group_id | lookup itsi_notable_group_system_lookup _key OUTPUT mod_time | lookup itsi_notable_group_user_lookup _key OUTPUT owner severity status | search NOT status=* AND mod_time=* | eval owner=index_owner, severity=index_severity, status=index_status, object_type="notable_group_user" | fields - index_owner, index_severity, index_status {noformat} If results look correct append the following Template:Outputlookup command and re-run search: {noformat}| outputlookup itsi_notable_group_user_lookup append=true key_field=itsi_group_id{noformat} This search should ideally update these Episodes: "2a617192-1858-4219-aba8-ed7b777f3035"
"ad3ec87e-05c2-4b1c-8ca9-c854ac6f6725"
"ccfa9689-a4e8-460e-a001-45e6891361a8" |
2022-08-09 | ITSI-25749 | Vital metrics data doesn't populate when there are more than 100 entities in ITSI |
2022-07-12 | ITSI-24964 | ITSI Searches ("Date Range", "Date & Time Range") do not honor auto-generated values; new Real-time search option fails (tstats not supported in a real-time search) Workaround: For Date & Time selections: manually enter/replace any portion of the auto-filled date for both start and end dates (even if replacing with the same value); or, select date from the calendar dropdown. For Date selection only: No workaround found so far. |
2022-05-03 | ITSI-23408 | Expiring ITSI licenses don't revert ITSI to ITE-W when the Splunk license is expired |
2022-05-02 | ITSI-23325 | Upgrade Readiness Dashboard can take over a minute to load. Workaround: To find the results of the most recent precheck, use this search with the time range set to 24 hours (or the interval of the precheck job if it has been customized): {noformat}index=_internal source=*itsi_migration_utility.log sourcetype=itsi_internal_log transaction_id {noformat} The search results should show the results of 14 prechecks and the object ids of any objects that fail any of the prechecks. |
2021-09-01 | ITSI-18709 | ITSI redirects to suite_redirect 500 Internal Server Error - because of python library isolation between apps Workaround: Step 1: Identify all the splunklib directories within the splunk apps directory using command find . -name 'splunklib' | xargs -r ls -lah .
Step 2: For each directory listed in step 1, check if file Step 3: Copy the Step 4: Clean the cached files using Step 5: Restart Splunk on the ITE Work or ITSI search head. |
2019-05-30 | ITSI-3322 | If you add a correlation search in ITSI which contains a sub-search returning into an eval, you get a message "Invalid search string: This search cannot be parsed when parse_only is set to true." Workaround: You can't use a sub-search returning into an eval in a correlation search. As a workaround, create and save a basic correlation search with all of the information you want outside of the search. Then as an admin user, go to Settings > Searches, reports, and alerts and open the correlation search you just created. Add the sub-search you were trying to add there. |
All ITSI Modules
Publication date | Issue number | Description |
---|---|---|
2017-03-21 | ITOA-7585 | When you bulk add services and an error caused by the racing condition occurs, the incorrect message "itsi_module does not exist" is displayed. |
2017-03-07 | MOD-979 | KPIs do not have consistent backfill settings across all modules. |
2017-01-17 | MOD-452 | The Analyze KPI button on the Service Details page is broken. |
2017-01-17 | MOD-402 | The Export to PDF option does not work in the drilldown to a module. |
2017-01-17 | MOD-296 | The extendable tab XML generator REST endpoint is located in DA-ITSI-OS instead of in common components where it can be used by all modules. |
2017-01-17 | MOD-591 | ITSI displays a misleading error message when a KPI template contains a field that cannot be resolved. |
2017-01-17 | MOD-498 | There is no upper limit to the number of characters a KPI title or description can contain. Long strings can negatively affect performance. |
2017-01-17 | MOD-309 | The Gruntfile.js included in ITSI modules uses double quotes instead of single quotes, which does not conform to the standard for all JavaScript files. |
2017-04-17 | MOD-2002 | When you drilldown from the Events tab, an "Invalid earliest_time" error occurs.
|
2017-01-17 | MOD-439 | Some modules do not have descriptions for saved searches. |
Application Server Module
Publication date | Issue number | Description |
---|---|---|
2017-01-27 | MOD-492 | If you reuse the same panel within a dashboard, the duplicate panel does not display any event data. |
Cloud Services Module
There are no known issues for this release.
Database Module
Publication date | Issue number | Description |
---|---|---|
2017-01-17 | MOD-586 | When a lookup is not configured for TA-Microsoft-SqlServer, ITSI displays a misleading error message on the server drilldown page. |
End User Experience Module
There are no known issues for this release.
Load Balancer Module
Publication date | Issue number | Description |
---|---|---|
2017-01-27 | MOD-492 | If you reuse the same panel within a dashboard, the duplicate panel does not display any event data. |
Operating System Module
Publication date | Issue number | Description |
---|---|---|
2017-04-13 | MOD-555 | The Storage Free Space % base search runs every minute while the Linux df command runs every 5 minutes. This causes data gaps. |
2017-04-10 | MOD-1964 | Windows data for memory free space is collected at different intervals than the Memory Free % KPI. |
2017-01-17 | MOD-1398 | Line, stack, and area charts do not display a metric gap when no metrics are available during a time period. |
Storage Module
There are no known issues for this release.
Virtualization Module
There are no known issues for this release.
Web Server Module
Publication date | Issue number | Description |
---|---|---|
2017-03-17 | MOD-320 | Some KPI ad hoc searches transform data with the stats command and do not retain time fields. The KPIs do not render anything and do not show thresholding details.
|
2017-03-17 | MOD-538 | When you add a new tab with panels and refresh the page, the page breaks. |
Fixed issues in Splunk IT Service Intelligence | Removed features in Splunk IT Service Intelligence |
This documentation applies to the following versions of Splunk® IT Service Intelligence: 4.13.0
Feedback submitted, thanks!