Splunk® Data Stream Processor

Function Reference

On April 3, 2023, Splunk Data Stream Processor will reach its end of sale, and will reach its end of life on February 28, 2025. If you are an existing DSP customer, please reach out to your account team for more information.
This documentation does not apply to the most recent version of Splunk® Data Stream Processor. For documentation on the most recent version, go to the latest release.

Union

This topic describes how to use the function in the Splunk Data Stream Processor.

Description

Combines streams with the same input schema into one stream with all of the events of the input streams. If the combined streams do not have the same schema, an error is shown.

Function Input/Output Schema

Function Input
collection<collection<record<R>>>
This function takes in multiple data streams where each stream has schema R.
Function Output
collection<record<R>>
This function outputs a single data stream with schema R.

Syntax

The required syntax is in bold.

union
<DataStream> ["," <DataStream>...]

Required arguments

DataStream
Syntax: <expression>
Description: The data stream you want to perform the union on.

Usage

If you are using the Union function in the Canvas Builder, click on the View Configuration button and select the function on the data stream branch that you'd like to union.
If you are using the Union function in the SPL2 Builder, you'll need to use SPL2 variables.

Using SPL2 variables to union data streams in the SPL2 Pipeline Builder

When constructing a pipeline using the SPL2 Pipeline Builder and performing a union, you must use SPL2 variables to properly union your data streams. SPL2 variables allow you to store branches of your data streams as SPL2 statements. Variable names must begin with a dollar sign ($) and can only contain letters, numbers, or underscores. You must have the same number of variables as branches in your data stream. For example, if you want to union three branches together, you will need three SPL2 variables.

You can store data streams as SPL2 variables and union these data streams like this:

$my_variable1 = | from read_from_aws_cloudwatch_metrics("4e1a64d8-0849-4324-9298-1e655ea4ba87") | eval sourcetype="cloudwatch"; 
$my_variable2 = | from receive_from_forwarders("forwarders:all") | eval sourcetype="forwarders";

|from $my_variable1 | union $my_variable2 | into write_index("", "metrics");

In this example, you have two variables representing the two data stream branches in your pipeline. The first variable stores the first branch of your pipeline containing a Read from AWS CloudWatch Metrics source function and an eval function. The second variable stores the second branch of your pipeline containing a Receive from Splunk Forwarders source function and a different eval function. After defining your variables, you can then reference them in your SPL2 statements and union the two branched data streams together. UnionUI.png

SPL2 example

This example unions together two branches, one branch containing the data from the Ingest Service and another branch containing the data from the Forwarders Service.

$forwarder_events = | from receive_from_forwarders("forwarders:all");
$ingest_api_events = | from receive_from_ingest_rest_api("rest-api:all");
| from $forwarder_events | union $ingest_api_events | into write_index("", "main");
Last modified on 17 July, 2020
To Splunk JSON   Where

This documentation applies to the following versions of Splunk® Data Stream Processor: 1.1.0


Was this topic useful?







You must be logged into splunk.com in order to post comments. Log in now.

Please try to keep this discussion focused on the content covered in this documentation topic. If you have a more general question about Splunk functionality or are experiencing a difficulty with Splunk, consider posting a question to Splunkbase Answers.

0 out of 1000 Characters