Ingest Processor is currently released as a preview only and is not officially supported. See Splunk General Terms for more information. For any questions on this preview, please reach out to ingestprocessor@splunk.com.

Route subsets of data using Ingest Processor

You can create a pipeline that routes a desired subset of incoming data so that it gets sent to a different destination.

Creating a pipeline that leverages the route command involves doing the following:

• Defining the routing criteria.
• Specifying the data destination for the routed data.

As a best practice for preventing unwanted data loss, make sure to always have a default destination for your Ingest Processor. Otherwise, all unprocessed data is dropped.

Prerequisites

Before starting to create a pipeline, confirm the following:

• The source type of the data that you want the pipeline to process is listed on the Source types page of your tenant. If your source type is not listed, then you must add that source type to your tenant and configure event breaking and merging definitions for it. See Add source types for Ingest Processor for more information for more information.
• The destination that you want the pipeline to send data to is listed on the Destinations page of your tenant. If your destination is not listed, then you must add that destination to your tenant. See Add or manage destinations for more information.

Route, thru, and branch SPL2 commands

See the following table to learn the differences between the route, thru and branch SPL2 commands.

SPL2 command	Description	SPL2 reference page and examples
branch	The SPL2 branch command processes one set of events or search results, in parallel, simultaneous pipelines. Each pipeline branch must end with the into command. Using the SPL2 branch command, you can take one set of data and run multiple pipelines simultaneously against that data. The simultaneous pipelines are referred to as branches. The results of the pipelines are placed into separate lookup or splv1sink datasets, using the into command. The data that you search can be events or search results.	branch command overview
route	Use the SPL2 route command in an SPL2 pipeline to route a desired subset of incoming data so that it gets sent to a different destination.	route command overview
thru	The SPL2 thru command writes data to a writeable dataset and then passes the same data to the next command in the search string. By default, the thru command appends data to the dataset.	thru command overview

Steps

Perform the following steps to create a pipeline that route subsets of data to different destinations:

1. Create a pipeline.
2. Configure routing in your pipeline.
3. Preview, save, and apply your pipeline.

Create a pipeline

Complete these steps to create a pipeline that receives data associated with a specific source type, optionally processes it, and sends that data to a destination.

1. Navigate to the Pipelines page and then select New pipeline.
2. Select Blank pipeline, and then select Next.
3. On the Define your pipeline's partition page, do the following:
1. Select the plus () icon next to Partition, or select the option that matches how you would like to create your partition in the Suggestions section.
2. In the Field field, specify the event field that you want the partitioning condition to be based on.
3. To specify whether the pipeline includes or excludes the data that meets the criteria, select Keep or Remove.
4. In the Operator field, select an operator for the partitioning condition.

In the Value field, enter the value that your partition should filter by to create the subset. Then select Apply. You can create as many conditions for a partition in a pipeline by selecting the plus () icon. Once you have defined your partition, select Next.

4. Select Next to confirm these definitions.
5. (Optional) On the Add sample data page, enter or upload sample data for generating previews that show how your pipeline processes data.

   The sample data must be in the same format as the actual data that you want to process. See Getting sample data for previewing data transformations for more information.

6. Select Next to confirm the sample data that you want to use for your pipeline.
7. On the Select destination dataset page, select the name of the destination that you want to send data to, and then select Next.

   If you're sending data to a Splunk platform deployment, be aware that the destination index is determined by a precedence order of configurations.

8. (Optional) Specify a target index as a field on each event. Click Done when complete.

You now have a simple pipeline that receives data for a specific source type and sends that data to a destination. In the next section, you'll configure this pipeline to route a subset of data to a different destination.

Configure routing in your pipeline

During the previous step, you added a source type to your pipeline, making the pipeline automatically filter for that source type in the incoming data that it receives. The next step is to configure the pipeline to route a subset of the received data to a different destination.

To specify whether your pipeline routes a subset of data to a different destination, do the following:

2. Select the plus icon () in the Actions section, then select Route data.
3. In the Field field, specify the event field that you want the partitioning condition to be based on.
4. To specify whether the pipeline includes or excludes the data that meets the criteria, select Keep or Remove.
5. In the Operator field, select an operator for the partitioning condition.
6. In the Value field, enter the value that your partition should filter by to create the subset. Then select Apply.
7. To select a data destination for the Route action, navigate to the Route action, and select Send data to $destination2.
8. In the Select data destination menu, select the name of the destination that you want to send data to. For example, the following pipeline filters the received data for events that have the sourcetype field set to Buttercup_Games and then routes those events to a data destination that is different from the rest of the pipeline:

   $pipeline = from $source 
   | route sourcetype == "Buttercup_Games", [ | into $destination2] 
   | into $destination;
   

9. Click Apply when complete.

You now have a pipeline that routes a subset of data to a destination that is from the rest of your pipeline's partition. In the next section, you'll verify that this pipeline processes data in the way that you expect and save it to be applied to an Ingest Processor pipeline.

Preview, save, and apply your pipeline

1. (Optional) Select the Preview Pipeline icon () to generate a preview that shows what your data looks like when it passes through the pipeline. For a pipeline with multiple destinations, run the pipeline preview then select the name of the destination that you want to check in the Preview drop-down list.
2. To save your pipeline, do the following:
1. Select Save pipeline.
2. In the Name field, enter a name for your pipeline.
3. (Optional) In the Description field, enter a description for your pipeline.
4. Select Save.

The pipeline is now listed on the Pipelines page, and you can apply it as needed.

3. To apply this pipeline, do the following:
1. Navigate to the Pipelines page.
2. In the row that lists your pipeline, select the Actions icon () and then select Apply.
3. Select the pipeline that you want to apply, and then select Save.

It can take a few minutes for the Ingest Processor service to finish applying your pipeline. During this time, all applied pipelines enter the Pending status. Once the operation is complete, the Pending Apply status icon () stops displaying beside the pipeline. Refresh your browser to check if the icon no longer displays.

Your applied pipeline can now route data so that only a desired subset of that data gets sent to the destination specified in the pipeline.