Splunk® Enterprise

Search Manual

Acrobat logo Download manual as PDF


Splunk Enterprise version 8.1 will no longer be supported as of April 19, 2023. See the Splunk Software Support Policy for details. For information about upgrading to a supported version, see How to upgrade Splunk Enterprise.
Acrobat logo Download topic as PDF

Search normalization

When you use the search or where command in a search string, the SPL processor might reorder the expression statement that follows the command for normalization purposes. The SPL processor applies two kinds of normalization logic to search strings: predicate flip and predicate sort.

For more information about predicates and predicate-based search optimization, see Built-in optimization.

Use the Job Inspector to see the results of search normalization and optimization. See Analyze search optimizations.

Benefits of search normalization

Some search optimizations perform better after search normalization. For example, the predicate merge optimization cannot merge where statements that place a field value before a field name. However, if the SPL processor applies predicate flip normalization to those statements so that the field name precedes the field value, the predicate merge optimization can merge it.

See Predicate merge.

Predicate flip normalization

Under predicate flip normalization, the SPL processor takes where statements that have field-value pairs where field values are placed ahead of field names and switches them so that the field names come first.

For example, in this search, the field value has been placed ahead of the field:

index=main | where "error"=status

After normalization, the field name and field value are flipped:

index=main | where (status == "error")

Predicate flip normalization only works when the SPL processor can distinguish the field name from the field value. The SPL processor puts numeric field values and string field values that are surrounded by quotes on the right side of the operator. When it is possible, the SPL processor flips value-field combinations where the value includes functions, such as value()=field.

The SPL processor will not apply predicate flip normalization to boolean, time, and IPv4 fields. For example, with a boolean value-field pair like true=purchased, the SPL processor cannot distinguish whether true or purchased is the field name.

Predicate sort normalization

Under predicate sort normalization, the SPL processor uses lexicographical sorting logic to ensure that search expressions and where statements are consistently ordered in the same way.

Predicate sort for the search command

When you use the search command in a string, the SPL processor applies predicate sort normalization to any boolean expressions that follow it.

For example, the following three searches use the search command with a boolean expression. These searches look different, but they produce the same result:

| search ( z OR y AND d AND c AND b AND a )

| search ( d AND z OR y AND c AND b AND a )

| search ( d AND ( z OR y ) AND ( c AND b AND a ) )

After normalization, those strings are reordered so that they share the following form:

| search ((y OR z) a b c d)

Predicate sort for the where command

When you use the where command in a string, the SPL processor applies predicate sort normalization to any boolean or arithmetic statements that follow it.

For example, these where statements have mathematical expressions that all resolve to the same result, but are ordered differently:

| where x = (d+(c-a)+c*b)*b

| where b*(d+(c-a)+c*b) = x

| where ((b*c)+d+(c-a))*b = x

After normalization, these where statements share the following form:

| where (x == ((((b * c) + (c - a)) + d) * b))

Example combining predicate flip and predicate sort

The following example combines predicate flip and predicate sort. Before normalization, you can have the following where statements:

| where status="error" OR code=500

| where "error"=status OR code=500

| where 500=code OR "error"=status

After normalization, these where statements share the following form:

| where ((code == 500) OR (status == "error"))

Disable search normalization

If you put your search expressions and where statements in a specific order for search performance reasons, you might want to disable search normalization. Predicate flip normalization and predicate sort normalization are controlled by separate settings in the limits.conf file. For reference, see the .spec file at limits.conf. You can disable one kind of normalization and leave the other enabled.

Splunk Cloud Platform
To disable search normalization, request help from Splunk Support. If you have a support contract, file a new case using the Splunk Support Portal at Support and Services. Otherwise, contact Splunk Customer Support.
Splunk Enterprise
To disable search normalization, follow these steps.
Prerequisites

Never change or copy the configuration files in the default directory. The files in the default directory must remain intact and in their original location. Make changes to the files in the local directory.

Disable predicate flip normalization
  1. Open or create a local limits.conf file at $SPLUNK_HOME/etc/system/local.
  2. Under the [search_optimization::search_flip_normalization] stanza, set enabled=false.
Disable predicate sort normalization
  1. Open or create a local limits.conf file at $SPLUNK_HOME/etc/system/local.
  2. Under the [search_optimization::search_sort_normalization] stanza, set enabled=false.
Last modified on 01 March, 2022
PREVIOUS
Built-in optimization
  NEXT
Control search execution using directives

This documentation applies to the following versions of Splunk® Enterprise: 7.0.0, 7.0.2, 7.0.3, 7.0.4, 7.0.5, 7.0.6, 7.0.7, 7.0.8, 7.0.9, 7.0.10, 7.0.11, 7.0.13, 7.1.0, 7.1.1, 7.1.2, 7.1.3, 7.1.4, 7.1.5, 7.1.6, 7.1.7, 7.1.8, 7.1.9, 7.1.10, 7.2.0, 7.2.2, 7.2.3, 7.2.4, 7.2.5, 7.2.6, 7.2.7, 7.2.8, 7.2.9, 7.2.10, 7.3.0, 7.3.1, 7.3.2, 7.3.3, 7.3.4, 7.3.5, 7.3.6, 7.3.7, 7.3.8, 7.3.9, 8.0.0, 8.0.1, 8.0.2, 8.0.3, 8.0.5, 8.0.10, 7.2.1, 7.0.1, 8.0.4, 8.0.9, 8.1.0, 8.1.1, 8.1.2, 8.1.3, 8.1.4, 8.1.5, 8.1.6, 8.1.7, 8.1.8, 8.1.9, 8.1.10, 8.1.11, 8.1.12, 8.1.13, 8.1.14, 8.2.0, 8.2.1, 8.2.2, 8.2.3, 8.2.4, 8.2.5, 8.2.6, 8.2.7, 8.2.8, 8.2.9, 8.2.10, 8.2.11, 8.2.12, 9.0.0, 9.0.1, 9.0.2, 9.0.3, 9.0.4, 9.0.5, 9.0.6, 9.0.7, 9.0.8, 9.1.0, 9.1.1, 9.1.2, 9.1.3, 9.2.0, 8.0.6, 8.0.7, 8.0.8


Was this documentation topic helpful?


You must be logged into splunk.com in order to post comments. Log in now.

Please try to keep this discussion focused on the content covered in this documentation topic. If you have a more general question about Splunk functionality or are experiencing a difficulty with Splunk, consider posting a question to Splunkbase Answers.

0 out of 1000 Characters