Splunk® Enterprise

Release Notes

Increased skipped search rate after upgrade to 9.0

Customers who have alerts configured that use the trigger condition "rises by" or "drops by" might see a significant increase in skipped or deferred searches after they upgrade to Splunk Enterprise version 9.0.0 through 9.0.5.

Run the following search from any search head in the cluster to determine if these alerts are configured:

| rest /servicesNS/-/-/saved/searches splunk_server=local f="title" f="eai*" f="alert_comparator" f="disabled" | search alert_comparator IN ("*rises*", "*drop*") | table eai:acl.app, eai:acl.owner, disabled, title, alert_comparator

If you use Splunk Enterprise version 9.0.0 through 9.0.5, you can determine if you are encountering this issue by looking for long running subsearches (200s or more) from the following alerts:

index=_audit sourcetype=audittrail action=search info=completed subsearch_AlertActionsRequredFields | rex field=search "savedsearch\=\"(?<user>\S+)\;(?<app>\S+)\;(?<savedsearch_name>.+)\"" | table _time host total_run_time search user app savedsearch_name | sort -total_run_time

On Splunk Enterprise version 9.0.0 through 9.0.5, you can avoid the issue by disabling the alerts or removing the trigger conditions from these alerts.

This issue is fixed in Splunk Enterprise version 9.0.6 and higher and in Splunk Enterprise version 9.1.0 and higher.

After upgrading to a fixed version, you must turn on allow_concurrent_dispatch_savedsearch in server.conf on the search heads. It is off by default. This change takes effect after a rolling restart of the search head cluster.

allow_concurrent_dispatch_savedsearch = 1

Last modified on 15 September, 2023
Known issues   Splunk Enterprise and anti-virus products

This documentation applies to the following versions of Splunk® Enterprise: 8.2.0, 8.2.1, 8.2.2, 8.2.3, 8.2.4, 8.2.5, 8.2.6, 8.2.7, 8.2.8, 8.2.9, 8.2.10, 8.2.11, 8.2.12, 9.0.0, 9.0.1, 9.0.2, 9.0.3, 9.0.4, 9.0.5, 9.0.6, 9.0.7, 9.0.8, 9.0.9, 9.0.10, 9.1.0, 9.1.1, 9.1.2, 9.1.3, 9.1.4, 9.1.5, 9.2.0, 9.2.1, 9.2.2

Was this topic useful?

You must be logged into splunk.com in order to post comments. Log in now.

Please try to keep this discussion focused on the content covered in this documentation topic. If you have a more general question about Splunk functionality or are experiencing a difficulty with Splunk, consider posting a question to Splunkbase Answers.

0 out of 1000 Characters