Workaround for search optimization issues
In 6.5.0, Splunk added a number of search optimizations. A few issues have been identified as a result of these optimizations.
Incorrect search results when search terms include a calculated field
Search optimization moves filters from the right to the left in a search string.
A search uses the
index=_internal | where user="admin"
The search optimization changes this search to the following search.
The reason for this optimization is that the
where command is case-sensitive and the
search command, which is implied before the pipe, is not case-sensitive.
However, if there is a calculated field present for the
user field, the optimized search is turned into this lispy expression:
[ AND index::_internal admin case ]
This search is looking for
index=_internal AND admin AND case. This is no field called
case and the search returns the incorrect search results.
This problem applies to both
field=CASE() and the
As a result, the optimized search returns zero results even though there are plenty of events in
There are several workarounds, depending on your situation.
Your fields are not based on indexed tokens
- If you have a field that is not based on indexed tokens, for example you are extracting part of a word, you might search for it like this:
index=_internal | search my_field="value"
- The search optimization converts the search string into this search:
index=_internal my_field="value" [ AND index::_internal value ]
- This will silently return incorrect search results because there is no indexed token value.
- Regardless of the optimizer changing the results, you should use the
fields.conffile to properly declare that this field is not an indexed token. Then the optimized results will be correct again and you will not have to use the piped search yourself.
Turning off optimization for a specific search
- If specific searches are not working since you upgraded to version 6.5.0, can turn off optimization for those searches.
- At the end of your search string, add this clause.
... |noop search_optimization=false
Turning off optimization for all searches
- This option is not recommended. Splunk is actively working on solutions to these optimization issues.
- If you cannot disable specific searches, or if the problem is pervasive, you can turn off the search optimization in the
In a distributed environment, this setting should be made on the search head.
- If you are using Splunk Cloud and want to turn off search optimization, open a Support ticket.
- Only users with the admin role, or a role with equivalent permissions, can change the search optimization settings for all searches.
- Review the steps in How to edit a configuration file in the Admin Manual.
- Open the local
limits.conffile for the Search app. For example,
- Under the [search_optimization::predicate_merge] stanza, set
Never change or copy the configuration files in the default directory. The files in the default directory must remain intact and in their original location.
The backslash character causes search errors
The backslash ( \ ) character is often used to escape a character in a search. In 6.5.0, when the character is used without quotation marks, an error is returned.
A search uses the backslash character to escape square bracket ( [ ] ) characters.
The search optimization is ignoring the backslash characters.
The resulting search is interpreted as a subsearch and returns errors.
Add quotation marks sound the expression.
This preserves the escaping characters and results in the intended search to look in the
_audit index and locate anything inside square brackets.
Linux kernel memory overcommitting and Splunk crashes
This documentation applies to the following versions of Splunk® Enterprise: 6.5.1, 6.5.2, 6.5.3, 6.5.4, 6.5.5, 6.5.6, 6.5.7