Ingest Processor pipeline syntax
A pipeline is a Search Processing Language, version 2 (SPL2) statement that specifies what data to process, how to process it, and what destination to send the processed data to. To configure Ingest Processor to transform and route data, you must create a pipeline and apply it.
The Ingest Processor solution supports a subset of SPL2 commands and functions. When writing a pipeline, you can include only certain commands and functions that are part of the ingestProcessor
profile. Additionally, when using a command or function that uses regular expressions, you must write the regular expressions in Regular Expression 2 (RE2) syntax. For information about the SPL2 and regular expression features that are supported for Ingest Processor pipelines, see the following sections on this page:
- SPL2 syntax for Ingest Processor pipelines
- SPL2 commands for Ingest Processor pipelines
- SPL2 evaluation functions for Ingest Processor pipelines
- SPL2 operators for Ingest Processor pipelines
- Regular expression syntax for Ingest Processor pipelines
For more information about profiles, see the SPL2 compatibility profiles in the SPL2 Search Reference.
The pipeline builder includes the SPL to SPL2 conversion tool, which you can use to convert SPL into SPL2 that is valid for Ingest Processor pipelines. See SPL to SPL2 Conversion tool in the SPL2 Search Reference.
SPL2 syntax for Ingest Processor pipelines
To create a Ingest Processor pipeline, you must write an SPL2 statement that uses the following syntax:
$pipeline = | from $source | <processing command> | into $destination;
The following table explains how each part of this pipeline syntax determines the data processing behavior by the Ingest Processor that the pipeline is applied to:
Pipeline syntax | Required? | Description |
---|---|---|
$pipeline
|
Yes | A variable indicating that the current SPL2 statement is a pipeline instead of a search query. |
from $source
|
Yes | A command that instructs the Ingest Processor to process all the incoming data that is associated with a given source type. |
<processing command> | No | One or more SPL2 commands that determine how the Ingest Processor handles data. Each command must be delimited by a pipe ( | ).
See the following sections on this page for more information:
If you don't include any processing commands, the Ingest Processor sends data to the destination without changing the data. |
into $destination
|
Yes | A command that instructs Ingest Processor to send data to a given destination. |
SPL2 commands for Ingest Processor pipelines
You can use the following SPL2 commands in your pipelines.
Each pipeline must include the from
and into
commands, as described in the SPL2 syntax for Ingest Processor pipelines section on this page. You can optionally use the other commands in this list as processing commands in your pipeline.
- branch
- eval
- expand
- fields
- flatten
- from
- into
- mvexpand
- rename
- rex
When writing regular expressions in an Ingest Processor pipeline, you must use RE2 syntax. See Regular expression syntax for Ingest Processor pipelines for more information.
- route
- thru
- where
SPL2 evaluation functions for Ingest Processor pipelines
You can use the following evaluation functions in your eval
and where
commands:
Type of function | Evaluation function |
---|---|
Comparison and conditional functions | |
Conversion functions | |
Cryptographic functions | |
Date and time functions | |
Informational functions | |
JSON functions | |
Mathematical functions | |
Multivalue eval functions | |
Statistical eval functions | |
Text functions | |
Trig and hyperbolic functions |
SPL2 operators for Ingest Processor pipelines
You can use the following operators in your processing commands:
Type of operator | Operator | Documentation |
---|---|---|
Relational operators |
|
See Relational operators in the SPL2 Search Manual. |
Logical operators |
|
See Logical operators in the SPL2 Search Manual. |
Arithmetic operators |
|
See Arithmetic operators in the SPL2 Search Reference. |
Other operators | IN | See IN operator in the SPL2 Search Manual. |
Regular expression syntax for Ingest Processor pipelines
Unlike the Splunk platform, the Ingest Processor solution supports Regular Expression 2 (RE2) syntax instead of Perl Compatible Regular Expressions (PCRE) syntax. When you use a command or function that uses regular expressions, such as the rex
command or the replace
evaluation function, you must write the regular expressions in RE2 syntax.
To learn about RE2 syntax and the differences between RE2 and PCRE, see https://github.com/google/re2/wiki/Syntax in the Google RE2 repository on GitHub.
Additionally, be aware that named capture groups must be written using this format:
(?P<fieldName>regular expression capture pattern)
Named capture groups are typically used in SPL2 statements to extract data values from the event body into fields. For example, to create an Ingest Processor pipeline that extracts HTTP status codes from the event body into a field named httpcode
, you need to include the following rex
command and named capture group:
rex field=_raw /(?P<httpcode>[1-5][0-9][0-9])/
System requirements for Ingest Processor | Create pipelines for Ingest Processor |
This documentation applies to the following versions of Splunk Cloud Platform™: 9.1.2308, 9.1.2312, 9.2.2403, 9.2.2406 (latest FedRAMP release)
Feedback submitted, thanks!