Add source types for Edge Processors
In the Edge Processor service, source type configurations are used to do the following:
- Break and merge the inbound stream of data into distinct events.
- Specify what data a pipeline processes.
If the source type that you want to work with is not included by default, then you can add and configure it in the Edge Processor service.
When you add a source type, you configure the following options. Each option is equivalent to a property that is supported in props.conf files:
Edge Processor option | props.conf property |
---|---|
Line breaking | LINE_BREAKER |
Merge lines into events | SHOULD_LINEMERGE |
Multiline event delimiter | BREAK_ONLY_BEFORE |
Maximum lines per event | MAX_EVENTS |
For more information about props.conf, see props.conf in the Splunk Enterprise Admin Manual.
If you already have a props.conf file with the configurations that you want to use, you might be able to reuse those configurations by copy-pasting them into the source types in the Edge Processor service. However, you must ensure that all of the regular expressions in these configurations use valid Regular Expression 2 (RE2) syntax. For more information about the regular expression syntax that the Edge Processor solutions supports, see Regular expression syntax for Edge Processor pipelines.
This is step 3 of 6 for using an Edge Processor to process data and route it to a destination. To see an overview of all of the steps, see Quick start: Process and route data using Edge Processors.
Prerequisites
Before you can add a source type in the Edge Processor service, you must know the exact name of the source type that you want to work with. This source type name must be identical to the value of the sourcetype
field in the data that you want to process and configure event breaking for.
Steps
- On the Source types page, select New source type.
- In the Name field, enter the exact name of the source type that you want to work with. The source type name must meet these requirements:
- The name must be unique. If you want to override a source type configuration that already exists in your tenant, you must either edit the existing source type configuration or rename it so that you can define a new configuration using the original source type name.
- The name cannot be
splunk-edge-processor-metrics
orsplunk-edge-processor-log
. These are reserved for internal use only.
- In the Line breaking field, specify the delimiter that indicates the end of one event and the start of another. If using a line break as the delimiter meets your requirements, then leave this field at the default value of
([\r\n]+)
. Otherwise, enter a different RE2 capture group that matches the delimiter.This delimiter gets dropped from your data. It is treated as something that exists between events rather than something that is part of an event. For more information, see the description of the
LINE_BREAKER
property in props.conf in the Splunk Enterprise Admin Manual. - If your inbound data consists of multiline events, then do the following:
- Select Merge lines into events.
- In the Multiline event delimiter field, enter an RE2 expression that matches the start of each multiline event.
- (Optional) To specify the maximum number of lines to include in a single multiline event, expand Advanced settings and enter your desired maximum number of lines in the Maximum lines per event field.
- (Optional) To generate a preview that shows how your source type configuration breaks and merges inbound data streams into events, do the following:
- Select the Edit sample data icon ().
- In the Edit sample data dialog box, enter or upload sample data for generating the preview.
The sample data must be in the same format as the actual data that is associated with the source type. See Getting sample data for previewing data transformations for more information.
- Select Save.
- Select the Run To Preview Source Type icon () to generate the preview. Use the preview results to validate your source type configuration.
- To save your source type, do the following:
- Select Save source type.
- (Optional) In the Description field, enter a description for the source type.
- Select Save.
You now have a source type configuration that breaks and merges any inbound data that has a matching sourcetype
value.
You can also use this source type in your pipelines. For information about creating pipelines and applying them to Edge Processors, see Create pipelines for Edge Processors and Apply pipelines to Edge Processors.
Using source types to break and merge data in Edge Processors | Edit, clone, or delete source types for Edge Processors |
This documentation applies to the following versions of Splunk Cloud Platform™: 9.0.2209, 9.0.2303, 9.0.2305, 9.1.2308, 9.1.2312, 9.2.2403 (latest FedRAMP release), 9.2.2406
Feedback submitted, thanks!