Ingest Processor pipeline syntax

A pipeline is a Search Processing Language, version 2 (SPL2) module containing a $pipeline that specifies what data to process, how to process it, and what destination to send the processed data to. To configure Ingest Processor to transform and route data, you must create a pipeline and apply it.

For information about modules and statements as SPL2 concepts, see Modules and SPL2 statements in the SPL2 Search Manual.

The Ingest Processor solution supports a subset of SPL2 commands and functions. When writing a pipeline, you can include only certain commands and functions that are part of the ingestProcessor profile. Additionally, when using a command or function that uses regular expressions, you must write the regular expressions in Regular Expression 2 (RE2) syntax. For information about the SPL2 and regular expression features that are supported for Ingest Processor pipelines, see the following sections on this page:

• SPL2 syntax for Ingest Processor pipelines
• SPL2 commands for Ingest Processor pipelines
• SPL2 evaluation functions for Ingest Processor pipelines
• SPL2 operators for Ingest Processor pipelines
• Regular expression syntax for Ingest Processor pipelines

For more information about profiles, see the SPL2 compatibility profiles in the SPL2 Search Reference.

The pipeline builder includes the SPL to SPL2 conversion tool, which you can use to convert SPL into SPL2 that is valid for Ingest Processor pipelines. See SPL to SPL2 Conversion tool in the SPL2 Search Reference.

SPL2 syntax for Ingest Processor pipelines

To create a Ingest Processor pipeline, you must write an SPL2 statement that uses the following syntax:

$pipeline = | from $source | <processing command> | into $destination;

The following table explains how each part of this pipeline syntax determines the data processing behavior by the Ingest Processor that the pipeline is applied to:

Pipeline syntax	Required?	Description
$pipeline	Yes	A variable indicating that the current SPL2 statement is a pipeline instead of a search query.
from $source	Yes	A command that instructs the Ingest Processor to process all the incoming data that is associated with a given source type.
<processing command>	No	One or more SPL2 commands that determine how the Ingest Processor handles data. Each command must be delimited by a pipe ( | ). See the following sections on this page for more information: SPL2 commands for Ingest Processor pipelines SPL2 evaluation functions for Ingest Processor pipelines SPL2 operators for Ingest Processor pipelines Regular expression syntax for Ingest Processor pipelines If you don't include any processing commands, the Ingest Processor sends data to the destination without changing the data.
into $destination	Yes	A command that instructs Ingest Processor to send data to a given destination.

SPL2 commands for Ingest Processor pipelines

You can use the following SPL2 commands in your pipelines.

Each pipeline must include the from and into commands, as described in the SPL2 syntax for Ingest Processor pipelines section on this page. You can optionally use the other commands in this list as processing commands in your pipeline.

• branch
• eval
• expand
• fields
• flatten
• from
• into
• lookup

Before you can use the lookup command in an Ingest Processor pipeline, you must import a lookup dataset into the pipeline. See Importing datasets into Ingest Processor pipelines on this page and Enrich data with lookups using Ingest Processor for more information.

• mvexpand
• rename
• rex

  When writing regular expressions in an Ingest Processor pipeline, you must use RE2 syntax. See Regular expression syntax for Ingest Processor pipelines for more information.

• route
• thru
• where

SPL2 evaluation functions for Ingest Processor pipelines

You can use the following evaluation functions in your eval and where commands:

Type of function	Evaluation function
Comparison and conditional functions	case cidrmatch coalesce if in like match When writing regular expressions in an Ingest Processor pipeline, you must use RE2 syntax. See Regular expression syntax for Ingest Processor pipelines for more information.
Conversion functions	ipmask tojson tonumber tostring
Cryptographic functions	md5 sha1 sha256 sha512
Date and time functions	relative_time strftime strptime time
Informational functions	isbool isint isnotnull isnull isnum isstr
JSON functions	json_append json_array json_array_to_mv json_delete json_extend json_extract json_extract_exact json_keys json_object json_set json_set_exact json_valid filter map reduce
Mathematical functions	abs exp ln log pi pow sqrt
Multivalue eval functions	mvappend mvcount mvdedup mvfind mvindex mvjoin mvrange mv_to_json_array split
Statistical eval functions	random
Text functions	len lower ltrim replace When writing regular expressions in an Ingest Processor pipeline, you must use RE2 syntax. See Regular expression syntax for Ingest Processor pipelines for more information. rtrim spath substr trim upper
Trig and hyperbolic functions	acos acosh asin asinh atan atan2 atanh cos cosh hypot sin sinh tan tanh

SPL2 operators for Ingest Processor pipelines

You can use the following operators in your processing commands:

Type of operator	Operator	Documentation
Relational operators	equals ( = ) or ( == ) does not equal ( != ) is greater than ( > ) is greater than or equal to ( >= ) is less than ( < ) is less than or equal to ( <= )	See Relational operators in the SPL2 Search Manual.
Logical operators	AND OR NOT XOR	See Logical operators in the SPL2 Search Manual.
Arithmetic operators	+ - / * %	See Arithmetic operators in the SPL2 Search Reference.
Other operators	IN	See IN operator in the SPL2 Search Manual.

Regular expression syntax for Ingest Processor pipelines

The Ingest Processor solution supports Perl Compatible Regular Expressions 2 (PCRE) syntax. All Regular Expression 2 (RE2) syntax must be replaced with PCRE syntax. All Regular Expression 2 (RE2) syntax must be replaced with PCRE syntax. When you use a command or function that uses regular expressions, such as the rex command or the replace evaluation function, you must write the regular expressions in PCRE2 syntax.

To learn about RE2 syntax and the differences between RE2 and PCRE, see https://github.com/google/re2/wiki/Syntax in the Google RE2 repository on GitHub.

Additionally, be aware that named capture groups must be written using this format:

(?P<fieldName>regular expression capture pattern)

Named capture groups are typically used in SPL2 statements to extract data values from the event body into fields. For example, to create an Ingest Processor pipeline that extracts HTTP status codes from the event body into a field named httpcode, you need to include the following rex command and named capture group:

rex field=_raw /(?P<httpcode>[1-5][0-9][0-9])/

Importing datasets into Ingest Processor pipelines

When configuring a pipeline, you can import datasets for the pipeline to use by adding an import statement before the $pipeline statement.

For example, you can import a lookup dataset, and then use the lookup command to configure a pipeline that enriches the received data with additional information from the lookup dataset. In this case, your pipeline would be an SPL2 module that contains the following 2 statements:

import <dataset_name> from <namespace>

$pipeline = | from $source | lookup <dataset_name> <lookup_field> AS <event_field> | into $destination

The namespace contains the name of your Ingest Processor tenant. The tenant name, dataset name, and field names must be enclosed in single quotation marks ( ' ) if either of the following conditions are true:

• The name begins with anything other than a-z, A-Z, or the underscore ( _ ) character.
• The name contains anything other than a-z, A-Z, 0-9, or the underscore ( _ ) character.

If you configure your lookup using the Enrich events with lookup action in the pipeline builder, the pipeline builder automatically adds both the import statement and lookup command to your pipeline.

When you add an import statement, you are not physically importing an item into the SPL2 module. Instead, you are adding a pointer to an item that resides in another module or namespace. When you use the item in a statement, the item is retrieved from the source. For more information, see Importing module items and datasets using SPL2 in the SPL2 Search Manual