Route subsets of data using an Edge Processor

You can create a pipeline that routes a desired subset of incoming data so that it gets sent to a different destination.

Creating a pipeline that leverages the route command involves doing the following:

• Defining the routing criteria.
• Specifying the data destination for the routed data.

As a best practice for preventing unwanted data loss, make sure to always have a default destination for your Edge Processor. Otherwise, all unprocessed data is dropped.

Prerequisites

Before starting to create a pipeline, confirm the following:

• The source type of the data that you want the pipeline to process is listed on the Source types page of your tenant. If your source type is not listed, then you must add that source type to your tenant and configure event breaking and merging definitions for it. See Add source types for Edge Processors for more information.
• The destination that you want the pipeline to send data to is listed on the Destinations page of your tenant. If your destination is not listed, then you must add that destination to your tenant. See Add or manage destinations for more information.

Steps

Perform the following steps to create a pipeline that route subsets of data to different destinations:

1. Create a pipeline.
2. Configure routing in your pipeline.
3. Preview, save, and apply your pipeline.

Create a pipeline

Complete these steps to create a pipeline that receives data associated with a specific source type, optionally processes it, and sends that data to a destination.

1. Navigate to the Pipelines page and then select New pipeline.
2. Select Blank pipeline, and then select Next.
3. On the Define your pipeline's partition page, do the following:
1. Select the plus () icon next to Partition, or select the option that matches how you would like to create your partition in the Suggestions section.
2. In the Field field, specify the event field that you want the partitioning condition to be based on.
3. To specify whether the pipeline includes or excludes the data that meets the criteria, select Keep or Remove.
4. In the Operator field, select an operator for the partitioning condition.

In the Value field, enter the value that your partition should filter by to create the subset. Then select Apply. You can create more conditions for a partition in a pipeline by selecting the plus () icon. Once you have defined your partition, select Next.

4. Select Next to confirm these definitions.
5. (Optional) On the Add sample data page, enter or upload sample data for generating previews that show how your pipeline processes data.

   The sample data must be in the same format as the actual data that you want to process. See Getting sample data for previewing data transformations for more information.

6. Select Next to confirm the sample data that you want to use for your pipeline.
7. Select the name of the destination that you want to send data to. Then, do the following:
1. If you selected a Splunk platform S2S or Splunk platform HEC destination, select Next.
2. If you selected another type of destination, select Done and skip the next step.
8. (Optional) If you're sending data to a Splunk platform deployment, you can specify a target index:
   1. In the Index name field, select the name of the index that you want to send your data to.
   2. (Optional) In some cases, incoming data already specifies a target index. If you want your Index name selection to override previous target index settings, then select the Overwrite previously specified target index check box.
   3. Select Done.

   Be aware that the destination index is determined by a precedence order of configurations. See How does an Edge Processor know which index to send data to? for more information.

You now have a simple pipeline that receives data for a specific source type and sends that data to a destination. In the next section, you'll configure this pipeline to route a subset of data to a different destination.

Configure routing in your pipeline

During the previous step, you added a source type to your pipeline, making the pipeline automatically filter for that source type in the incoming data that it receives.

The next step is to configure the pipeline to route a subset of the received data to a different destination.

To specify whether your pipeline routes a subset of data to a different destination, do the following:

1. Select the the plus icon () in the Actions section, then select the Route data option.
2. In the Field field, specify the event field that you want the partitioning condition to be based on.
3. To specify whether the pipeline includes or excludes the data that meets the criteria, select Keep or Remove.
4. In the Operator field, select an operator for the partitioning condition.
5. In the Value field, enter the value that your partition should filter by to create the subset. Then select Apply.
6. To select a data destination for the Route action, navigate to the Route action, and select Send data to $destination2.
7. In the Select data destination menu, select the name of the destination that you want to send data to. For example, the following pipeline filters the received data for events that have the sourcetype field set to Buttercup_Games and then routes those events to a data destination that is different from the rest of the pipeline:

   $pipeline = from $source 
   | route sourcetype == "Buttercup_Games", [ | into $destination2] 
   | into $destination;
   

8. Click Apply when complete.

You now have a pipeline that routes a subset of data to a destination that is from the rest of your pipeline's partition. In the next section, you'll verify that this pipeline processes data in the way that you expect and save it to be applied to a Edge Processor pipeline.

Preview, save, and apply your pipeline

1. (Optional) Select the Preview Pipeline icon () to generate a preview that shows what your data looks like when it passes through the pipeline. For a pipeline with multiple destinations, run the pipeline preview then select the name of the destination that you want to check in the Preview drop-down list.
2. To save your pipeline, do the following:
   1. Select Save pipeline.
   2. In the Name field, enter a name for your pipeline.
   3. (Optional) In the Description field, enter a description for your pipeline.
   4. Select Save.

   If your pipeline is valid, the Edge Processor service prompts you to apply it to an Edge Processor.

3. To apply this pipeline to an Edge Processor, do the following:
   1. In the Apply pipeline prompt, select Yes, apply.
   2. Select the Edge Processors that you want to apply the pipeline to, and then select Save.

   You can only apply pipelines to Edge Processors that are in the Healthy status.

   It can take a few minutes for the Edge Processor service to finish applying your pipeline to an Edge Processor. During this time, the affected Edge Processors enter the Pending status. To confirm that the process completed successfully, do the following:

   • Navigate to the Edge Processors page. Then, verify that the Instance health column for the affected Edge Processors shows that all instances are back in the Healthy status.
   • Navigate to the Pipelines page. Then, verify that the Applied column for the pipeline contains a The pipeline is applied icon ().

The Edge Processor that you applied the pipeline to can now route different subsets of the received data to different destinations.