Add source types for Ingest Processor

In Ingest Processor, source type configurations are used to do the following:

• Break and merge the inbound stream of data from all_data_ready into distinct events.
• Specify what data a pipeline processes. For information on how source types determine the subset of the incoming data that a pipeline processes, see Partitions.

If the source type that you want to work with is not included by default, then you can add and configure it in Ingest Processor.

When you add a source type, you configure the following options. Each option is equivalent to a property that is supported in props.conf files:

Ingest Processor option	props.conf property
Line breaking	LINE_BREAKER
Merge lines into events	SHOULD_LINEMERGE
Multiline event delimiter	BREAK_ONLY_BEFORE
Maximum lines per event	MAX_EVENTS

If you already have a props.conf file with the configurations that you want to use, you can reuse those configurations by copy-pasting them into the source types in the Ingest Processor service. For more information about props.conf, see props.conf in the Splunk Enterprise Admin Manual.

Prerequisites

Before you can add a source type in the Ingest Processor service, you must know the exact name of the source type that you want to work with. This source type name must be identical to the value of the sourcetype field in the data that you want to process and configure event breaking for.

Steps

1. On the Source types page, select New source type.
2. In the Name field, enter the exact name of the source type that you want to work with. The source type name must meet these requirements:
• The name must be unique. If you want to override a source type configuration that already exists in your tenant, you must either edit the existing source type configuration or rename it so that you can define a new configuration using the original source type name.
• The name cannot be splunk-data-processor-metrics or splunk-data-processor-log. These are reserved for internal use only.
3. In the Line breaking field, specify the delimiter that indicates the end of one event and the start of another. If using a line break as the delimiter meets your requirements, then leave this field at the default value of ([\r\n]+). Otherwise, enter a different Regular Expression 2 (RE2) capture group that matches the delimiter.

   This delimiter gets dropped from your data. It is treated as something that exists between events rather than something that is part of an event. For more information, see the description of the LINE_BREAKER property in props.conf in the Splunk Enterprise Admin Manual.

4. If your inbound data consists of multiline events, then do the following:
1. Select Merge lines into events.
2. In the Multiline event delimiter field, enter an RE2 expression that matches the start of each multiline event.
3. (Optional) To specify the maximum number of lines to include in a single multiline event, expand Advanced settings and enter your desired maximum number of lines in the Maximum lines per event field.
5. (Optional) To generate a preview that shows how your source type configuration breaks and merges inbound data streams into events, do the following:
1. Select the Edit sample data icon ().
2. In the Edit sample data dialog box, enter or upload sample data for generating the preview. The sample data must be in the same format as the actual data that is associated with the source type.
3. Select Save.
4. Select the Run To Preview Source Type icon () to generate the preview. Use the preview results to validate your source type configuration.
6. To save your source type, do the following:
1. Select Save source type.
2. (Optional) In the Description field, enter a description for the source type.
3. Select Save.

You now have a source type configuration that breaks and merges any inbound data that has a matching sourcetype value. Additionally, when creating pipelines, you can select this source type to specify the subset of the incoming data that you want the pipeline to process.

You can also use this source type in your pipelines. For information about creating pipelines and applying them to Ingest Processor, see Create pipelines for Ingest Processor and Apply a pipeline.