Splunk Cloud Platform

Use Edge Processors

Add source types for Edge Processors

In the Edge Processor service, source type configurations are used to do the following:

  • Break and merge the inbound stream of data into distinct events.
  • Specify what data a pipeline processes.

If the source type that you want to work with is not included by default, then you can add and configure it in the Edge Processor service.

When you add a source type, you configure the following options. Each option is equivalent to a property that is supported in props.conf files:

Edge Processor option props.conf property
Line breaking LINE_BREAKER
Merge lines into events SHOULD_LINEMERGE
Multiline event delimiter BREAK_ONLY_BEFORE
Maximum lines per event MAX_EVENTS

For more information about props.conf, see props.conf in the Splunk Enterprise Admin Manual.

If you already have a props.conf file with the configurations that you want to use, you might be able to reuse those configurations by copy-pasting them into the source types in the Edge Processor service. However, you must ensure that all of the regular expressions in these configurations use valid Regular Expression 2 (RE2) syntax. For more information about the regular expression syntax that the Edge Processor solutions supports, see Regular expression syntax for Edge Processor pipelines.

This is step 3 of 6 for using an Edge Processor to process data and route it to a destination. To see an overview of all of the steps, see Quick start: Process and route data using Edge Processors.

This diagram shows an overview of the steps required to set up and use an Edge Processor.

Prerequisites

Before you can add a source type in the Edge Processor service, you must know the exact name of the source type that you want to work with. This source type name must be identical to the value of the sourcetype field in the data that you want to process and configure event breaking for.

Steps

  1. On the Source types page, select New source type.
  2. In the Name field, enter the exact name of the source type that you want to work with. The source type name must meet these requirements:
    • The name must be unique. If you want to override a source type configuration that already exists in your tenant, you must either edit the existing source type configuration or rename it so that you can define a new configuration using the original source type name.
    • The name cannot be splunk-edge-processor-metrics or splunk-edge-processor-log. These are reserved for internal use only.
  3. In the Line breaking field, specify the delimiter that indicates the end of one event and the start of another. If using a line break as the delimiter meets your requirements, then leave this field at the default value of ([\r\n]+). Otherwise, enter a different RE2 capture group that matches the delimiter.

    This delimiter gets dropped from your data. It is treated as something that exists between events rather than something that is part of an event. For more information, see the description of the LINE_BREAKER property in props.conf in the Splunk Enterprise Admin Manual.

  4. If your inbound data consists of multiline events, then do the following:
    1. Select Merge lines into events.
    2. In the Multiline event delimiter field, enter an RE2 expression that matches the start of each multiline event.
    3. (Optional) To specify the maximum number of lines to include in a single multiline event, expand Advanced settings and enter your desired maximum number of lines in the Maximum lines per event field.
  5. (Optional) To generate a preview that shows how your source type configuration breaks and merges inbound data streams into events, do the following:
    1. Select the Edit sample data icon (Image of the Edit sample data icon).
    2. In the Edit sample data dialog box, enter or upload sample data for generating the preview.

      The sample data must be in the same format as the actual data that is associated with the source type. See Getting sample data for previewing data transformations for more information.

    3. Select Save.
    4. Select the Run To Preview Source Type icon (This image shows an icon with a triangle pointing right.) to generate the preview. Use the preview results to validate your source type configuration.
  6. To save your source type, do the following:
    1. Select Save source type.
    2. (Optional) In the Description field, enter a description for the source type.
    3. Select Save.

You now have a source type configuration that breaks and merges any inbound data that has a matching sourcetype value.

You can also use this source type in your pipelines. For information about creating pipelines and applying them to Edge Processors, see Create pipelines for Edge Processors and Apply pipelines to Edge Processors.

Last modified on 26 April, 2024
Using source types to break and merge data in Edge Processors   Edit, clone, or delete source types for Edge Processors

This documentation applies to the following versions of Splunk Cloud Platform: 9.0.2209, 9.0.2303, 9.0.2305, 9.1.2308, 9.1.2312, 9.2.2403 (latest FedRAMP release), 9.2.2406


Was this topic useful?







You must be logged into splunk.com in order to post comments. Log in now.

Please try to keep this discussion focused on the content covered in this documentation topic. If you have a more general question about Splunk functionality or are experiencing a difficulty with Splunk, consider posting a question to Splunkbase Answers.

0 out of 1000 Characters