The sequence of search-time operations

When you run a search, Splunk software runs several operations to derive various knowledge objects and apply them to the events returned by the search. These knowledge objects include extracted fields, calculated fields, lookup fields, field aliases, tags, and event types.

Splunk software performs these operations in a specific sequence. This sequence can cause problems if you configure something at the top of the process order with a definition that references the result of a configuration that is farther down in the process order.

Search-time operation sequence

The following table presents the search-time operation sequence as a list. After the list you can find more information about each operation in the sequence, as well as examples of the search-time operations order.

Each operation can have configurations that reference fields derived by operations that precede them in the sequence. However, those same configurations cannot contain fields that are derived by operations that follow them in the sequence.

You can configure most of these operations through Splunk Web, although some configuration options are available only by making manual edits to configuration files. Make all manual file-based operation configurations on the search-head tier.

This list does not include index-time operations, such as default and indexed field extraction. Index-time operations precede all search-time operations. See Index-time versus search time in the Splunk Enterprise Managing Indexers and Clusters of Indexers manual.

Search-time operation order	Operation name	Configurable in Splunk Web?	Location of file configuration
1	Inline field extraction (no field transform)	Yes	EXTRACT-<class> in a stanza in the props.conf file.
2	Field extraction that uses a field transform	Yes	REPORT-<class> in a stanza in the props.conf file.
3	Automatic key-value field extraction	No	In stanzas in the props.conf file, where KV_MODE is set to a valid value other than none. If no KV_MODE value is specified for a stanza, it is set to auto by default.
4	Field filters	Yes	fieldFilterName stanza in the field_filters.conf file.
5	Field aliasing	Yes	FIELDALIAS-<class> in a stanza in the props.conf file.
6	Calculated fields	Yes	EVAL-<fieldname> in a stanza in the props.conf file.
7	Lookups	Yes	LOOKUP-<class> in a stanza in the props.conf file.
8	Event types	Yes	In a stanza in the eventtypes.conf file.
9	Tags	Yes	In a stanza in the tags.conf file.

See About configuration files for an overview of configuration file usage.

Example of search-time operations order

Consider calculated fields. Calculated field operations are in the middle of the search-time operation sequence. Splunk software performs several other operations ahead of them, and it performs several more operations after them. Calculated fields derive new fields by running the values of fields that already exist in an event through an eval formula. This means that a calculated field formula cannot include fields in its formula that are added to your events by operations that follow it in the search-time operation sequence.

For example, when you design an eval expression for a calculated field, you can include extracted fields in the expression, because field extractions are processed at the start of the search-time operation sequence. By the time Splunk software processes calculated fields, the field extractions exist and the calculated field operation can complete correctly.

However, make sure an eval expression for a calculated field never includes fields that are added through a lookup operation. Splunk software always performs calculated field operations ahead of lookup operations. This means that fields added through lookups at search time are unavailable when Splunk software processes calculated fields. You will get an error message if your calculated field eval expression includes fields that are added through lookups.

Inline field extractions

Inline field extractions are explicit field extractions that do not include a field transform reference. An explicit field extraction is a field extraction that is configured to extract a specific field or set of fields.

Each inline field extraction configuration is specific to events belonging to a particular host, source, or source type.

This operation does not include automatic key-value field extractions. Automatic key-value field extractions are their own operation category.

Splunk Web management

To create and manage inline field extractions, follow these steps:

1. In Splunk Web, go to Settings.
2. Navigate to Settings > Fields > Field extractions.
3. Create a new field extraction or open an existing field extraction.

You can also use the field extractor utility to design inline field extractions.

Configure inline field extractions

Create EXTRACT-<class> configurations within props.conf file stanzas.

Restrictions

Splunk software processes all inline field extractions belonging to a specific host, source, or source type in lexicographical order according to their <class> value. This means that you cannot reference a field extracted by EXTRACT-aaa in the field extraction definition for EXTRACT-ZZZ, but you can reference a field extracted by EXTRACT-aaa in the field extraction definition for EXTRACT-ddd. See Lexicographical processing of field extraction configurations.

Because inline field extractions are near the top of the search-time operation sequence, they cannot reference fields that are derived and added to events by other search-time operations that come later.

See also

• See Build field extractions with the field extractor to create inline field extractions in Splunk Web. The field extractor does not require you to understand how to write regular expressions.
• See Use the Field Extractions page to create inline field extractions in Splunk Web using the Field Extractions page in Settings.
• See Create and maintain search-time field extractions through configuration files to configure inline field extractions in the props.conf file.

Field extraction that uses a field transform

Field extraction configurations that reference a field transform are always processed by Splunk software after it processes inline field extractions. Like inline field extractions, each transform-referencing field extraction is explicitly configured to extract a specific field or set of fields.

Each transform-referencing field extraction configuration is specific to events belonging to a particular host, source, or source type.

This operation does not include automatic key-value field extractions. Automatic key-value field extractions are their own operation category.

Splunk Web management

To create and manage field extractions that use field transforms, follow these steps:

1. In Splunk Web, go to Settings.
2. Navigate to Settings > Fields.
3. In the Field Extractions and Field Transformations pages, set up the field extraction.

Configure field extraction that uses a field transform

Create REPORT-<class> configurations within props.conf file stanzas. The REPORT-<class> configurations include a reference to an additional configuration in the transforms.conf file.

Restrictions

Splunk software processes all inline field extractions belonging to a specific host, source, or source type in lexicographical order according to their <class> value. This means that you cannot reference a field extracted by EXTRACT-aaa in the field extraction definition for EXTRACT-ZZZ, but you can reference a field extracted by EXTRACT-aaa in the field extraction definition for EXTRACT-ddd. See Lexicographical processing of field extraction configurations.

Transform-referencing field extraction configurations can reference fields that are extracted through inline field extraction operations. They cannot reference fields that are derived and added to events by automatic key-value field extractions and other operations that take place later in the search-time operation sequence.

See also

• See Use the field transformations page to create the transforms.conf part of a transform-referencing search-time field extraction.
• See Use the field extractions page to create the props.conf file part of a transform-referencing search-time field extraction.
• See Create and maintain search-time field extractions through configuration files to configure transform-referencing field extractions in transforms.conf file and props.conf files.
• See Extracting a field that was already extracted during inline field extraction to learn how to preserve values for fields that are extracted twice during the search-time operation sequence.

Automatic key-value field extraction

A field extraction configuration that uses the KV_MODE setting to automatically extract fields for events associated with a specific host, source, or source type.

Automatic key-value field extraction is not explicit in that you cannot configure it to find a specific field or set of fields. It looks for any key-value patterns in events that it can find and extracts them as field-value pairs. You can configure key-value field extraction to extract fields from structured data formats like JSON, CSV, and table-formatted events. You can also disable search-time key-value field extraction for specific hosts, sources, and source types.

Automatic key-value extraction always takes place after explicit field extraction methods, like in inline field extraction and transform--referencing field extraction.

Splunk Web management

You can configure the KV_MODE setting for source types through Splunk Web.

KV_MODE defaults to automatic key-value field extraction for all source types unless it is set to another value. For example, if you want to disable search-time key-value field extraction for a specific source type, you must set KV_MODE to none for that source type.

Here is how you can edit or update KV_MODE for a source type in Splunk Web.

1. In Splunk Web, go to Settings.
2. Navigate to Settings > Source types.
3. Locate a source type that you want to edit. Select Edit for that source type.
4. Select Advanced.
5. If KV _MODE is not among the settings listed for the source type you are editing, you can add it by selecting New setting and entering KV_MODE into the Name cell.
   

   When KV_MODE is not present it means that Splunk software applies the default for KV_MODE to the source type. KV_MODE is set to auto by default, which means that Splunk software runs automatic key-value field extraction at search time for any data with the source type.

6. Change the Value of KV_MODE as necessary. See Configure automatic key-value field extraction for configuration details.
7. Select Save to save your changes to the source type configuration.

If you need to disable JSON field extraction for a source type without disabling automatic key-value field extraction for the source type, you can use this method to add the AUTO_KV_JSON setting with a Value of false to the source type configuration.

For more information about editing source types with Splunk Web, see Manage source types in the Splunk Cloud Platform Getting Data In manual.

Configure automatic key-value field extraction

Set up automatic key-value field extractions for a specific host, source, or source type by finding or creating the appropriate stanza in the props.conf file and setting KV_MODE to auto, auto_escaped, multi, json, xml, or none.

When KV_MODE is not set for a props.conf file stanza, that stanza has KV_MODE=auto by default.

When KV_MODE is set to auto or auto_escaped, automatic JSON field extraction takes place alongside other automatic key-value field extractions. If you need to disable JSON field extraction without changing the KV_MODE value from auto, add AUTO_KV_JSON=false to the stanza. When not set, AUTO_KV_JSON defaults to true.

Restrictions

Splunk software processes automatic key-value field extractions in the order that it finds them in events.

See also

See Configure automatic key-value field extraction.

Field filters

Preview features are provided by Splunk to you "as is" without any warranties, maintenance and support, or service level commitments. Splunk makes this preview feature available in its sole discretion and may discontinue it at any time. Use of preview features is subject to the Splunk General Terms.

Field filters control the search results that are visible to users at search time. You can apply a field filter to all nonexempt roles, which then affects the results of searches run by users assigned with those roles. Field filters do not modify the source event in the index, but remove or replace specific indexed, _raw, search time, and default fields and their values at search time when those fields appear in search results. You can remove specific fields and their values by redacting them with a null value. Alternatively, you can redact the value of a specific field by replacing it with a custom string such as XXXX, or you can obfuscate the field value by replacing it with a hash using SHA-256 or SHA-512 (SHA-2 family) hash functions.

By default, to create, edit, or delete field filters, you must be a member of the admin or sc_admin role. To view field filters, you must be a member of the admin, sc_admin, or power user role. See Define roles on the Splunk platform with capabilities in Securing Splunk Platform.

Splunk Web management

To create and manage field filters using Splunk Web, follow these steps:

1. Select Settings > Field filters from Users And Authentication.
2. Select New field filter.

See Create field filters using Splunk Web in Securing the Splunk Platform.

Configuration file management

To create and manage field filters using configuration files, update the field_filters.conf file by adding action = <actions> to the <fieldFilterName> stanza for each field filter.

To maintain data integrity and security, do not use configuration files to exempt roles from field filters. Instead, use Splunk Web or Splunk platform REST API endpoints to exempt roles from field filters.

See Create field filters using configuration files in Securing the Splunk Platform.

Splunk platform REST API management

You can use the Splunk platform REST API to create and manage field filters.

See the following in the Splunk Cloud Platform REST API Reference Manual:

• authorization/fieldfilters
• authorization/fieldfilters/{name}
• authorization/roles
• authorization/roles/{name}

Restrictions

Since field filters are in the middle of the search-time operation sequence, field filters affect search-time operations that come later for fields that are filtered. For example, if the user field is replaced with XXX, then field aliasing for user extracts the value XXX instead of the user's name. This process has particular implications for downstream operations that depend on the value of the field that is changed by a field filter. If your searches look for particular fields that are added through operations such as calculated fields, lookups, or tags, or you search on specific event types, be aware that field filters that redact or obfuscate your fields can cause your searches to break. If you want to use these kinds of operations with a field that is filtered, configure field filters to replace the field value with a hash, which preserves the statistical uniqueness of the field and makes it available to operations that come later in the search-time sequence. Alternatively, you might need to re-evaluate search operations that are used together with field filters.

The following are operations that can be affected by field-value obfuscation and break existing searches when used with field filters:

Operation	Description
Calculated fields	The eval expression that calculates the field might break when it gets field values that are now XXX.
Lookups	Lookups add field-value combinations from lookup tables to event data and can break if Splunk software is unable to match field-value combinations in your event data with field-value combinations in external lookup tables.
Event types	The search that defines the event type might be looking to match field values that are now XXX.
tag command	If the value of a field for a tag's field-value pair is replaced with XXX, the tag is no longer applied.

See also

In Securing the Splunk Platform:

• Protect PII, PHI, and other sensitive data with field filters
• Define roles on the Splunk platform with capabilities

Field aliasing

Field aliasing is the application of field alias configurations, which enable you to reference a single field in a search by multiple alternate names, or aliases.

Each field alias configuration is specific to events belonging to a particular host, source, or source type.

Splunk Web management

To create and manage field aliases, follow these steps:

1. In Splunk Web, go to Settings.
2. Navigate to Settings > Fields > Field aliases.
3. Create a new field extraction or open an existing field alias.

Configure field aliasing

Create FIELDALIAS-<class> configurations in props.conf file stanzas.

Restrictions

Splunk software processes field aliases belonging to a specific host, source, or source type in lexicographical order. See Lexicographical processing of field extraction configurations.

You can create aliases for fields that are extracted at index time or search time. You cannot create aliases for fields that are added to events by search-time operations that follow the field aliasing process, like lookups and calculated fields.

See also

• Create field aliases in Splunk Web
• Configure field aliases with props.conf

Calculated fields

Configurations that create one or more fields through the calculation of eval expressions and add those fields to events. The eval expression can use values of fields that are already present in the event due to index-time or search-time field extraction processes.

Each calculated field configuration is specific to events belonging to a particular host, source, or source type.

Splunk Web management

To create and manage calculated fields, follow these steps:

1. In Splunk Web, go to Settings.
2. Navigate to Settings > Fields > Calculated fields.
3. Create a new calculated field or open an existing calculated field.

Configure calculated fields

Create calculated fields by adding EVAL-<fieldname> configurations to props.conf file stanzas.

Restrictions

All EVAL-<fieldname> configurations within a single props.conf stanza are processed in parallel instead of sequentially. This means you can't chain together calculated field expressions where the evaluation of one calculated field is used in the expression for the next calculated field.

Calculated fields can reference all types of field extractions. They can't reference lookups, event types, or tags.

You can't create a calculated field that is scoped to an aliased host, source, or source type. See Creation of a calculated field on an aliased source is not supported in the Knowledge Manager Manual.

See also

• About calculated fields
• Create calculated fields with Splunk Web
• Configure calculated fields with props.conf

Lookups

Configurations that add fields from lookup tables to events when the lookup table fields are matched with one or more fields already present in those events. There are four types of lookup configurations:

• CSV lookups
• External lookups
• KV store lookups
• Geospatial lookups

Each lookup configuration is specific to events belonging to a particular host, source, or source type.

Splunk Web management

To create and manage your lookups, follow these steps:

1. In Splunk Web, go to Settings.
2. Navigate to Settings > Lookups.
3. Create a new lookup definition or open an existing lookup definition.

Configure lookups

Define lookups that automatically add fields to events in search results by creating a LOOKUP-<class> configuration in the props.conf file. Each LOOKUP-<class> includes a reference to a [<lookup_name>] stanza in the transforms.conf file.

Restrictions

Splunk software processes lookups belonging to a specific host, source, or source type in lexicographical order. See Lexicographical processing of field extraction configurations.

Lookup configurations can reference fields that are added to events by field extractions, field aliases, and calculated fields. They cannot reference event types and tags.

See also

• About lookups

Event types

Configurations that add event type field-value pairs to events that match the search strings that define the event types.

Splunk Web management

After you run a search, save it as an event type. You can also define and maintain event types in Settings > Event types.

Configure event types

Configure event types in eventtypes.conf file stanzas.

Restrictions

Splunk software processes event types first by priority score and then by lexicographical order. So it processes all event types with a Priority of 1 first, and applies them to events in lexicographical order. Then it processes event types with a Priority of 2, and so on.

Search strings that define event types cannot reference tags. Event types are always processed and added to events before tags.

See also

• About event types
• Define event types in Splunk Web
• Automatically find and build event types
• Configure event types directly in eventtypes.conf

Tags

Configurations that add tags to specific field-value pairs in events.

Splunk Web management

You can add tags directly to field-value pairs in search results. You can also define and maintain tags in Settings > Tags.

Configure tags

Configure tags in tags.conf file stanzas.

Restrictions

Splunk software applies tags to field-value pairs in events in lexicographical order, first by the field value, and then by the field name. See Lexicographical processing of field extraction configurations.

You can apply tags to any field-value pair in an event, whether it is extracted at index time, extracted at search time, or added through some other method, such as an event type, lookup, or calculated field.

See also

• Tag field value pairs in Search
• Define and manage tags in Settings

Lexicographical processing of knowledge object configurations

Splunk software processes the following knowledge objects in lexicographical order, according to the host, source, or source type they belong to:

• Inline field extractions
• Field extractions that use a field transform
• Field aliases
• Event types, after they are sorted according to priority
• Lookups

Lexicographical order

Splunk software also processes tags in lexicographical order, but they are not associated with a specific host, source, or source type.

Lexicographical order sorts items based on the values used to encode the items in computer memory. In Splunk software, this is almost always UTF-8 encoding, which is a superset of ASCII.

• Numbers are sorted before letters. Numbers are sorted based on the first digit. For example, the numbers 10, 9, 70, 100 are sorted lexicographically as 10, 100, 70, 9.
• Uppercase letters are sorted before lowercase letters.
• Symbols are not standard. Some symbols are sorted before numeric values. Other symbols are sorted before or after letters.

Splunk software also uses lexicographical ordering to determine configuration file precedence among app directories. See Configuration file precedence in the Splunk Enterprise Admin Manual.

Example

Splunk software extracts inline field extractions to a specific host, source, or source type in ASCII sort order. This means that when it processes inline field extractions belonging to the access_combined_wcookies source type, it processes an extraction called REPORT-BBB before REPORT-ZZZ, then processes REPORT-ZZZ before REPORT-aaa, and so on.

This means that you cannot reference a field extracted by REPORT-aaa in the field extraction definition for REPORT-BBB.

For example, this configuration doesn't work because the first_ten field is extracted after the first_two field, due to the field extraction process ordering (aaa < ZZZ).

 [splunkd]
EXTRACT-aaa = ^(?<first_ten>.{10})
EXTRACT-ZZZ = (?<first_two>.{2}) in first_ten

The following configuration works because the first_ten field is extracted before the first_two field, due to the field extraction process ordering (ZZZ > mmm).

 [mongod]
EXTRACT-ZZZ = ^(?<first_ten>.{10})
EXTRACT-mmm = (?<first_two>.{2}) in first_ten

Here is a search you can use to verify these configuration issues.

index=_internal (sourcetype=splunkd OR sourcetype=mongod) | stats values(first_ten) values(first_two) by sourcetype

The process order within a single props.conf file

The Splunk Enterprise Admin Manual contains several topics about configuration file administration. One of these topics, Attribute precedence within a single props.conf file in the Splunk Enterprise Admin Manual, includes information about knowledge object processing order, as well as the following topics:

• Precedence between sets of stanzas affecting the same host, source, or source type.
• Overriding the default lexicographical order in the props.conf file.
• Precedence for events with multiple attribute assignments.