Workaround for search optimization issues

In 6.5.0, Splunk added a number of search optimizations. A few issues have been identified as a result of these optimizations.

Incorrect search results when search terms include a calculated field

Search optimization moves filters from the right to the left in a search string.

Example

A search uses the where command.

index=_internal | where user="admin"

The search optimization changes this search to the following search.

index=_internal user=CASE("admin")

Problem details

The reason for this optimization is that the where command is case-sensitive and the search command, which is implied before the pipe, is not case-sensitive.

However, if there is a calculated field present for the user field, the optimized search is turned into this lispy expression:

[ AND index::_internal admin case ]

This search is looking for index=_internal AND admin AND case. This is no field called case and the search returns the incorrect search results.

This problem applies to both field=CASE() and the field=TERM().

As a result, the optimized search returns zero results even though there are plenty of events in _internal for user="admin".

Workaround

There are several workarounds, depending on your situation.

Your fields are not based on indexed tokens

If you have a field that is not based on indexed tokens, for example you are extracting part of a word, you might search for it like this:

index=_internal | search my_field="value"

The search optimization converts the search string into this search:

index=_internal my_field="value" [ AND index::_internal value ]

This will silently return incorrect search results because there is no indexed token value.

Regardless of the optimizer changing the results, you should use the fields.conf file to properly declare that this field is not an indexed token. Then the optimized results will be correct again and you will not have to use the piped search yourself.

Turning off optimization for a specific search

If specific searches are not working since you upgraded to version 6.5.0, can turn off optimization for those searches.

At the end of your search string, add this clause.

... |noop search_optimization=false

Turning off optimization for all searches

This option is not recommended. Splunk is actively working on solutions to these optimization issues.

If you cannot disable specific searches, or if the problem is pervasive, you can turn off the search optimization in the limits.conf file.

In a distributed environment, this setting should be made on the search head.

If you are using Splunk Cloud and want to turn off search optimization, open a Support ticket.

Prerequisites

• Only users with the admin role, or a role with equivalent permissions, can change the search optimization settings for all searches.
• Review the steps in How to edit a configuration file in the Admin Manual.

Steps

1. Open the local limits.conf file for the Search app. For example, $SPLUNK_HOME/etc/apps/<app_name>/local.

Never change or copy the configuration files in the default directory. The files in the default directory must remain intact and in their original location.

2. Under the [search_optimization::predicate_merge] stanza, set enabled to false.

The backslash character causes search errors

The backslash ( \ ) character is often used to escape a character in a search. In 6.5.0, when the character is used without quotation marks, an error is returned.

Example

A search uses the backslash character to escape square bracket ( [ ] ) characters.

index=_audit \[*\]

The search optimization is ignoring the backslash characters.

index=_audit [*]

Problem details

The resulting search is interpreted as a subsearch and returns errors.

Workaround

Add quotation marks sound the expression.

index=_audit "\[*\]"

This preserves the escaping characters and results in the intended search to look in the _audit index and locate anything inside square brackets.