
Workaround for search optimization issues
In 6.5.0, Splunk added a number of search optimizations. A few issues have been identified as a result of these optimizations.
Incorrect search results when search terms include a calculated field
Search optimization moves filters from the right to the left in a search string.
Example
A search uses the where
command.
index=_internal | where user="admin"
The search optimization changes this search to the following search.
index=_internal user=CASE("admin")
Problem details
The reason for this optimization is that the where
command is case-sensitive and the search
command, which is implied before the pipe, is not case-sensitive.
However, if there is a calculated field present for the user
field, the optimized search is turned into this lispy expression:
[ AND index::_internal admin case ]
This search is looking for index=_internal AND admin AND case
. This is no field called case
and the search returns the incorrect search results.
This problem applies to both field=CASE()
and the field=TERM()
.
As a result, the optimized search returns zero results even though there are plenty of events in _internal
for user="admin"
.
Workaround
There are several workarounds, depending on your situation.
Your fields are not based on indexed tokens
- If you have a field that is not based on indexed tokens, for example you are extracting part of a word, you might search for it like this:
index=_internal | search my_field="value"
- The search optimization converts the search string into this search:
index=_internal my_field="value" [ AND index::_internal value ]
- This will silently return incorrect search results because there is no indexed token value.
- Regardless of the optimizer changing the results, you should use the
fields.conf
file to properly declare that this field is not an indexed token. Then the optimized results will be correct again and you will not have to use the piped search yourself.
Turning off optimization for a specific search
- If specific searches are not working since you upgraded to version 6.5.0, can turn off optimization for those searches.
- At the end of your search string, add this clause.
... |noop search_optimization=false
Turning off optimization for all searches
- This option is not recommended. Splunk is actively working on solutions to these optimization issues.
- If you cannot disable specific searches, or if the problem is pervasive, you can turn off the search optimization in the
limits.conf
file.
In a distributed environment, this setting should be made on the search head.
- If you are using Splunk Cloud and want to turn off search optimization, open a Support ticket.
- Prerequisites
- Only users with the admin role, or a role with equivalent permissions, can change the search optimization settings for all searches.
- Review the steps in How to edit a configuration file in the Admin Manual.
- Steps
- Open the local
limits.conf
file for the Search app. For example,$SPLUNK_HOME/etc/apps/<app_name>/local
. - Under the [search_optimization::predicate_merge] stanza, set
enabled
tofalse
.
Never change or copy the configuration files in the default directory. The files in the default directory must remain intact and in their original location.
The backslash character causes search errors
The backslash ( \ ) character is often used to escape a character in a search. In 6.5.0, when the character is used without quotation marks, an error is returned.
Example
A search uses the backslash character to escape square bracket ( [ ] ) characters.
index=_audit \[*\]
The search optimization is ignoring the backslash characters.
index=_audit [*]
Problem details
The resulting search is interpreted as a subsearch and returns errors.
Workaround
Add quotation marks sound the expression.
index=_audit "\[*\]"
This preserves the escaping characters and results in the intended search to look in the _audit
index and locate anything inside square brackets.
PREVIOUS Linux kernel memory overcommitting and Splunk crashes |
NEXT Timestamp recognition of dates with two-digit years fails beginning January 1, 2020 |
This documentation applies to the following versions of Splunk® Enterprise: 6.5.1, 6.5.2, 6.5.3, 6.5.4, 6.5.5, 6.5.6, 6.5.7, 6.5.8, 6.5.9, 6.5.10
Feedback submitted, thanks!