
Edge Processor pipeline syntax
A pipeline is a Search Processing Language, version 2 (SPL2) module containing a $pipeline
statement that specifies what data to process, how to process it, and what destination to send the processed data to. To configure an Edge Processor to process and send data, you must create pipelines and apply them to your Edge Processor. See System architecture and Create pipeline for Edge Processors for more information.
For information about modules and statements as SPL2 concepts, see Modules and SPL2 statements in the SPL2 Search Manual.
The Edge Processor solution supports a subset of SPL2 commands and functions. When creating a pipeline, you can use only the commands and functions that are part of the EdgeProcessor
profile. Additionally, when using a command or function that uses regular expressions, you must write the regular expressions in Regular Expression 2 (RE2) syntax. For information about the SPL2 and regular expression features that are supported for Edge Processor pipelines, see the following sections on this page:
- SPL2 syntax for Edge Processor pipelines
- SPL2 commands for Edge Processor pipelines
- SPL2 evaluation functions for Edge Processor pipelines
- SPL2 operators for Edge Processor pipelines
- Regular expression syntax for Edge Processor pipelines
For more information about profiles, see SPL2 compatibility profiles in the SPL2 Search Reference.
SPL2 syntax for Edge Processor pipelines
When you create an Edge Processor pipeline in the pipeline builder, you create an SPL2 statement that uses the following syntax:
$pipeline = | from $source | <processing command> | into $destination;
The following table explains how each part of this pipeline syntax determines the data processing behavior by the Edge Processor that the pipeline is applied to:
Pipeline syntax | Required? | Description |
---|---|---|
$pipeline
|
Yes | A variable indicating that the current SPL2 statement is a pipeline instead of a search query. |
from $source
|
Yes | A command that instructs the Edge Processor to process a subset of the incoming data. This subset is determined by the partition of the pipeline, which you configure using the pipeline builder. See Create pipelines for Edge Processors for more information. |
<processing command> | No | One or more SPL2 commands that determine how the Edge Processor handles data. Each command must be delimited by a pipe ( | ). See the following sections on this page for more information:
If you don't include any processing commands, the Edge Processor sends data to the destination without changing the data. |
into $destination
|
Yes | A command that instructs the Edge Processor to send data to a given destination. You specify the destination using the pipeline builder. See Create pipelines for Edge Processors for more information. |
SPL2 commands for Edge Processor pipelines
You can use the following SPL2 commands in your pipelines.
Each pipeline must include the from
and into
commands, as described in the SPL2 syntax for Edge Processor pipelines section on this page. You can optionally use the other commands in this list as processing commands in your pipeline.
- eval
- expand
- fields
- flatten
- from
- into
- lookup
Before you can use the
lookup
command in an Edge Processor pipeline, you must import a lookup dataset into the pipeline. See Importing datasets into Edge Processor pipelines on this page and Enrich data with lookups using an Edge Processor for more information. - mvexpand
- rename
- rex
When writing regular expressions in an Edge Processor pipeline, you must use RE2 syntax. See Regular expression syntax for Edge Processor pipelines on this page for more information.
- where
SPL2 evaluation functions for Edge Processor pipelines
You can use the following evaluation functions in your eval
and where
commands:
Type of function | Evaluation function |
---|---|
Comparison and conditional functions | |
Conversion functions | |
Cryptographic functions | |
Date and time functions | |
Informational functions | |
JSON functions | |
Multivalue eval functions | |
Statistical eval functions | |
Text functions | |
Trig and hyperbolic functions |
SPL2 operators for Edge Processor pipelines
You can use the following operators in your processing commands:
Type of operator | Operator | Documentation |
---|---|---|
Relational operators |
|
See Relational operators in the SPL2 Search Manual. |
Logical operators |
|
See Logical operators in the SPL2 Search Manual. |
Arithmetic operators |
|
See Arithmetic operators in the SPL2 Search Reference. |
Other operators | IN | See IN operator in the SPL2 Search Manual. |
Regular expression syntax for Edge Processor pipelines
Unlike the Splunk platform, the Edge Processor solution supports Regular Expression 2 (RE2) syntax instead of Perl Compatible Regular Expressions (PCRE) syntax. When you use a command or function that uses regular expressions, such as the rex
command or the replace
evaluation function, you must write the regular expressions in RE2 syntax.
To learn about RE2 syntax and the differences between RE2 and PCRE, see https://github.com/google/re2/wiki/Syntax in the Google RE2 repository on GitHub.
In particular, be aware that RE2 and PCRE accept different syntaxes for named capture groups. Named capture groups are typically used in SPL2 statements to extract data values from the event body into fields. The RE2 syntax for a named capture group is (?P<fieldName>regular expression capture pattern)
, while the PCRE syntax is (?<fieldName>regular expression capture pattern)
.
Example: Extracting fields in an Edge Processor pipeline
To create an Edge Processor pipeline that extracts HTTP status codes from the event body into a field named httpcode
, you need to include the following rex
command, which contains a named capture group written in RE2 syntax:
rex field=_raw /(?P<httpcode>[1-5][0-9][0-9])/
Example: Extracting fields in a search
To define a search that extracts HTTP status codes from the event body into a field named httpcode
, you need to include the following rex
command instead, which contains a named captured group written in PCRE syntax:
rex field=_raw /(?<httpcode>[1-5][0-9][0-9])/
Importing datasets into Edge Processor pipelines
When configuring a pipeline, you can import datasets for the pipeline to use by adding an import
statement before the $pipeline
statement.
For example, you can import a lookup dataset, and then use the lookup
command to configure a pipeline that enriches the received data with additional information from the lookup dataset. In this case, your pipeline would be an SPL2 module that contains the following 2 statements:
import <dataset_name> from <namespace> $pipeline = | from $source | lookup <dataset_name> <lookup_field> AS <event_field> | into $destination;
The namespace contains the name of your Edge Processor tenant. The tenant name, dataset name, and field names must be enclosed in single quotation marks ( ' ) if either of the following conditions are true:
- The name begins with anything other than a-z, A-Z, or the underscore ( _ ) character.
- The name contains anything other than a-z, A-Z, 0-9, or the underscore ( _ ) character.
When you add an import
statement, you are not physically importing an item into the SPL2 module. Instead, you are adding a pointer to an item that resides in another module or namespace. When you use the item in a statement, the item is retrieved from the source. For more information, see Importing module items and datasets using SPL2 in the SPL2 Search Manual.
PREVIOUS Configure global Edge Processor settings |
NEXT Create pipelines for Edge Processors |
This documentation applies to the following versions of Splunk Cloud Platform™: 9.0.2209, 9.0.2303, 9.0.2305 (latest FedRAMP release), 9.1.2308
Feedback submitted, thanks!