Create pipelines for Edge Processors

To specify how you want your Edge Processors to process and route your data, you must create pipelines and apply them to the Edge Processors.

This is step 5 of 6 for using an Edge Processor to process data and route it to a destination. To see an overview of all of the steps, see Quick start: Process and route data using Edge Processors.

A pipeline is a set of data processing instructions written in the Search Processing Language, version 2 (SPL2). To create a valid pipeline, you must complete the following tasks:

• Define the pipeline's partition, or the subset of data that you want this pipeline to process.
• Specify the destination that the pipeline sends processed data to.
• Write an SPL2 statement that defines what data to process, how to process it, and where to send the processed data to.
• Optionally, you can also add sample data to ensure your pipeline processes data as desired.

When you apply a pipeline to an Edge Processor, the Edge Processor uses those instructions to process the data that it receives.

Preventing data loss

Each pipeline filters the incoming data for a specified source type, and only processes data of that source type. Any data that is associated with a different source type is excluded from the pipeline. If the Edge Processor doesn't have an additional pipeline that accepts the excluded data, that data is either routed to the default destination or dropped.

As a best practice for preventing unwanted data loss, make sure to always have a default destination for your Edge Processors. Otherwise, all unprocessed data is dropped. See Add an Edge Processor.

Prerequisites

Before starting to create a pipeline, confirm the following:

• The source type of the data that you want the pipeline to process is listed on the Source types page of your tenant. If your source type is not listed, then you must add that source type to your tenant and configure event breaking and merging definitions for it. See Add source types for Edge Processors for more information.
• The destination that you want the pipeline to send data to is listed on the Destinations page of your tenant. If your destination is not listed, then you must add that destination to your tenant. See Add or manage destinations for more information.

Steps

Complete these steps to create a pipeline that receives data associated with a specific source type, optionally processes it, and sends that data to a destination.

1. Navigate to the Pipelines page and then select New pipeline.
2. Select Blank pipeline and then select Next.
3. Select or enter a sourcetype to define the subset of data you want this pipeline to process.
4. Select Next to confirm your partition.
5. (Optional) Enter or upload sample data for generating previews that show how your pipeline processes data.

   The sample data must be in the same format as the actual data that you want to process. See Getting sample data for previewing data transformations for more information.

6. Select Next to confirm your sample data.
7. Select the name of the destination that you want to send data to, and then select Done.




If you're sending data to a Splunk platform deployment, be aware that the destination index is determined by a precedence order of configurations. See How does an Edge Processor know which index to send data to? for more information.

8. (Optional) To process the incoming data before sending it to a destination, add processing commands to the SPL2 statement. For information and examples of the types of data processing actions that you can define in your pipeline, see the following pages:
   • Edge Processor pipeline syntax
   • Route internal logs from forwarders using an Edge Processor
   • Filter and mask data using an Edge Processor
   • Extract fields from event data using an Edge Processor

   Make sure that your pipeline contains one SPL2 statement only. Do not define multiple SPL2 statements in the same pipeline.

9. (Optional) To generate a preview of how your pipeline processes data based on the sample data that you provided earlier, select the Preview Pipeline icon (). Use the preview results to validate your pipeline configuration.
10. To save your pipeline, do the following:
    1. Select Save pipeline.
    2. In the Name field, enter a name for your pipeline.
    3. (Optional) In the Description field, enter a description for your pipeline.
    4. Select Save.

    The pipeline is now listed on the Pipelines page, and you can now apply it to Edge Processors as needed.

11. To apply this pipeline to an Edge Processor, do the following:
    1. Navigate to the Pipelines page.
    2. In the row that lists your pipeline, select the Actions icon () and then select Apply/Remove.
    3. Select the Edge Processors that you want to apply the pipeline to, and then select Save.

       You can only apply pipelines to Edge Processors that are in the Healthy status.

    It can take a few minutes for this process to be completed. During this time, the affected Edge Processors enter the Pending status. To confirm that the process completed successfully, do the following:

    • Navigate to the Edge Processors page. Then, verify that the Instance health column for the affected Edge Processors shows that all instances are back in the Healthy status.
    • Navigate to the Pipelines page. Then, verify that the Applied column for the pipeline contains a The pipeline is applied icon ().

    You might need to refresh your browser to see the latest updates.

    For information about other ways to apply pipelines to Edge Processors, see Apply pipelines to Edge Processors.

The Edge Processors that you applied the pipeline to can now process and route data as specified in the pipeline configuration.

Next step

After creating a pipeline and applying it to your Edge Processor, you can configure data sources to send data to your Edge Processor. See Get data from a forwarder into an Edge Processor and Get data into an Edge Processor using HTTP Event Collector.