Union

This topic describes how to use the function in the .

Description

Combines streams with the same input schema into one stream with all of the events of the input streams. If the combined streams do not have the same schema, an error is shown.

Function Input/Output Schema

Function Input

collection<collection<record<R>>>

This function takes in multiple data streams where each stream has schema R.

Function Output

collection<record<R>>

This function outputs a single data stream with schema R.

Syntax

The required syntax is in bold.

union

<DataStream> ["," <DataStream>...]

Required arguments

DataStream

Syntax: <expression>

Description: The data stream you want to perform the union on.

Usage

If you are using the Union function in the Canvas Builder, click on the View Configuration button and select the function on the data stream branch that you'd like to union.
If you are using the Union function in the SPL2 Builder, you'll need to use SPL2 variables.

Using SPL2 variables to union data streams in the SPL2 Pipeline Builder

When constructing a pipeline using the SPL2 Pipeline Builder and performing a union, you must use SPL2 variables to properly union your data streams. SPL2 variables allow you to store branches of your data streams as SPL2 statements. Variable names must begin with a dollar sign ($) and can only contain letters, numbers, or underscores. You must have the same number of variables as branches in your data stream. For example, if you want to union three branches together, you will need three SPL2 variables.

You can store data streams as SPL2 variables and union these data streams like this:

$my_variable1 = | from read_from_aws_cloudwatch_metrics("4e1a64d8-0849-4324-9298-1e655ea4ba87") | eval sourcetype="cloudwatch"; 
$my_variable2 = | from forwarders("forwarders:all") | eval sourcetype="forwarders";

|from $my_variable1 | union $my_variable2 | into index("", "metrics");

In this example, you have two variables representing the two data stream branches in your pipeline. The first variable stores the first branch of your pipeline containing a Amazon CloudWatch source function and an eval function. The second variable stores the second branch of your pipeline containing a Forwarders Service source function and a different eval function. After defining your variables, you can then reference them in your SPL2 statements and union the two branched data streams together.

SPL2 example

When working in the SPL View, you can write the function by providing the arguments in this exact order.

This example unions together two branches, one branch containing the data from the Ingest Service and another branch containing the data from the Forwarders Service.

$forwarder_events = | from forwarders("forwarders:all");
$ingest_api_events = | from ingest_rest_api("rest-api:all");
| from $forwarder_events | union $ingest_api_events | into index("", "main");