Search normalization

When you use the search or where command in a search string, the SPL processor might reorder the expression statement that follows the command for normalization purposes. The SPL processor applies two kinds of normalization logic to search strings: predicate flip and predicate sort.

For more information about predicates and predicate-based search optimization, see Built-in optimization.

Use the Job Inspector to see the results of search normalization and optimization. See Analyze search optimizations.

Benefits of search normalization

Some search optimizations perform better after search normalization. For example, the predicate merge optimization cannot merge where statements that place a field value before a field name. However, if the SPL processor applies predicate flip normalization to those statements so that the field name precedes the field value, the predicate merge optimization can merge it.

See Predicate merge.

Predicate flip normalization

Under predicate flip normalization, the SPL processor takes where statements that have field-value pairs where field values are placed ahead of field names and switches them so that the field names come first.

For example, in this search, the field value has been placed ahead of the field:

index=main | where "error"=status

After normalization, the field name and field value are flipped:

index=main | where (status == "error")

Predicate flip normalization only works when the SPL processor can distinguish the field name from the field value. The SPL processor puts numeric field values and string field values that are surrounded by quotes on the right side of the operator. When it is possible, the SPL processor flips value-field combinations where the value includes functions, such as value()=field.

The SPL processor will not apply predicate flip normalization to boolean, time, and IPv4 fields. For example, with a boolean value-field pair like true=purchased, the SPL processor cannot distinguish whether true or purchased is the field name.

Predicate sort normalization

Under predicate sort normalization, the SPL processor uses lexicographical sorting logic to ensure that search expressions and where statements are consistently ordered in the same way.

Predicate sort for the search command

When you use the search command in a string, the SPL processor applies predicate sort normalization to any boolean expressions that follow it.

For example, the following three searches use the search command with a boolean expression. These searches look different, but they produce the same result:

| search ( z OR y AND d AND c AND b AND a )

| search ( d AND z OR y AND c AND b AND a )

| search ( d AND ( z OR y ) AND ( c AND b AND a ) )

After normalization, those strings are reordered so that they share the following form:

| search ((y OR z) a b c d)

Predicate sort for the where command

When you use the where command in a string, the SPL processor applies predicate sort normalization to any boolean or arithmetic statements that follow it.

For example, these where statements have mathematical expressions that all resolve to the same result, but are ordered differently:

| where x = (d+(c-a)+c*b)*b

| where b*(d+(c-a)+c*b) = x

| where ((b*c)+d+(c-a))*b = x

After normalization, these where statements share the following form:

| where (x == ((((b * c) + (c - a)) + d) * b))

Example combining predicate flip and predicate sort

The following example combines predicate flip and predicate sort. Before normalization, you can have the following where statements:

| where status="error" OR code=500

| where "error"=status OR code=500

| where 500=code OR "error"=status

After normalization, these where statements share the following form:

| where ((code == 500) OR (status == "error"))

Disable search normalization

If you put your search expressions and where statements in a specific order for search performance reasons, you might want to disable search normalization. Predicate flip normalization and predicate sort normalization are controlled by separate settings in the limits.conf file. For reference, see the .spec file at limits.conf. You can disable one kind of normalization and leave the other enabled.

Splunk Cloud Platform

To disable search normalization, request help from Splunk Support. If you have a support contract, file a new case using the Splunk Support Portal at Support and Services. Otherwise, contact Splunk Customer Support.

Splunk Enterprise

To disable search normalization, follow these steps.

Prerequisites

• Only users with file system access, such as system administrators, can disable search normalization using configuration files.
• Review the steps in How to edit a configuration file in the Splunk Enterprise Admin Manual.
• You can have configuration files with the same name in your default, local, and app directories. Read Where you can place (or find) your modified configuration files in the Splunk Enterprise Admin Manual.

Never change or copy the configuration files in the default directory. The files in the default directory must remain intact and in their original location. Make changes to the files in the local directory.

Disable predicate flip normalization

1. Open or create a local limits.conf file at $SPLUNK_HOME/etc/system/local.
2. Under the [search_optimization::search_flip_normalization] stanza, set enabled=false.

Disable predicate sort normalization

1. Open or create a local limits.conf file at $SPLUNK_HOME/etc/system/local.
2. Under the [search_optimization::search_sort_normalization] stanza, set enabled=false.