Increased skipped search rate after upgrade to 9.0

Customers who have alerts configured that use the trigger condition "rises by" or "drops by" might see a significant increase in skipped or deferred searches after they upgrade to Splunk Enterprise version 9.0.0 through 9.0.5.

Run the following search from any search head in the cluster to determine if these alerts are configured:

| rest /servicesNS/-/-/saved/searches splunk_server=local f="title" f="eai*" f="alert_comparator" f="disabled" | search alert_comparator IN ("*rises*", "*drop*") | table eai:acl.app, eai:acl.owner, disabled, title, alert_comparator

If you use Splunk Enterprise version 9.0.0 through 9.0.5, you can determine if you are encountering this issue by looking for long running subsearches (200s or more) from the following alerts:

index=_audit sourcetype=audittrail action=search info=completed subsearch_AlertActionsRequredFields | rex field=search "savedsearch\=\"(?<user>\S+)\;(?<app>\S+)\;(?<savedsearch_name>.+)\"" | table _time host total_run_time search user app savedsearch_name | sort -total_run_time

On Splunk Enterprise version 9.0.0 through 9.0.5, you can avoid the issue by disabling the alerts or removing the trigger conditions from these alerts.

This issue is fixed in Splunk Enterprise version 9.0.6 and higher and in Splunk Enterprise version 9.1.0 and higher.

After upgrading to a fixed version, you must turn on allow_concurrent_dispatch_savedsearch in server.conf on the search heads. It is off by default. This change takes effect after a rolling restart of the search head cluster.

server.conf
[shclustering]
allow_concurrent_dispatch_savedsearch = 1