Splunk® Data Stream Processor

Connect to Data Sources and Destinations with DSP

Acrobat logo Download manual as PDF


Acrobat logo Download topic as PDF

Connecting multiple data sources to your DSP pipeline

When creating a data pipeline in the , you can choose to connect multiple data sources to the pipeline. For example, you can create a single pipeline that gets data from a Splunk forwarder, a syslog server, and Amazon Kinesis Data Streams concurrently. You can apply transformations to the data from all three data sources as the data passes through the pipeline, and then send the transformed data out from the pipeline to a destination of your choosing.

To connect a pipeline to multiple data sources, you can use either of the following methods:

Method Description
Start your data pipeline with the Splunk DSP Firehose source function. The Splunk DSP Firehose source function collects data from a subset of the data sources that DSP supports, and outputs the combined data in a single stream. Use this method if the Splunk DSP Firehose source function supports your data sources, and if you don't need to apply any transformations to the data from each data source before combining the streams.

See the rest of this page for more information about the Splunk DSP Firehose.

Configure a connection and a source function for each data source, and then use a union function to combine the data streams from these source functions into a single stream at the start of your pipeline. Use this method if the Splunk DSP Firehose function does not support your data sources, or if you want to apply specific transformations to the data streams before combining them. See Get data from Splunk DSP Firehose in the Function Reference manual to confirm if your data source is supported.

See Union in the Function Reference manual and the Building a pipeline chapter in the Use the manual for more information about using the union function.

Connecting to multiple data sources using the Splunk DSP Firehose

To connect to multiple data sources using the Splunk DSP Firehose, you must complete the following tasks:

  1. For each data source that you want to collect data from, create a connection. See Get data from Splunk DSP Firehose in the Function Reference manual to confirm that your data source is supported, and refer to the corresponding chapters in this Connect to Data Sources and Destinations with DSP manual for instructions on creating the connection.
  2. Create a pipeline that starts with the Splunk DSP Firehose source function.

When you activate the pipeline, the Splunk DSP Firehose source function starts collecting data from all the supported data sources that have a valid DSP connection. Each event or metric event is received into the pipeline as a record.

If any data fails to get into the pipeline, check the connection settings to make sure that you have the correct credentials.

Last modified on 18 December, 2020
PREVIOUS
Configure SC4S to send syslog data to DSP
  NEXT
Connecting multiple data destinations to your DSP pipeline

This documentation applies to the following versions of Splunk® Data Stream Processor: 1.2.0, 1.2.1


Was this documentation topic helpful?

You must be logged into splunk.com in order to post comments. Log in now.

Please try to keep this discussion focused on the content covered in this documentation topic. If you have a more general question about Splunk functionality or are experiencing a difficulty with Splunk, consider posting a question to Splunkbase Answers.

0 out of 1000 Characters