Connecting Microsoft Azure Event Hubs to your DSP pipeline as a data source

When creating a data pipeline in the , you can connect to Microsoft Azure Event Hubs and use it as a data source. You can get data from an event hub into a pipeline, transform the data as needed, and then send the transformed data out from the pipeline to a destination of your choosing.

To connect to Azure Event Hub as a data source, you must complete the following tasks:

1. If the event hub that you want to get data from does not have a consumer group that can be used solely by your DSP pipeline, create one. See the Using a dedicated consumer group for each pipeline section on this page for more information about this best practice.
2. Create a connection that allows DSP to access your Azure Event Hubs data. See Create a DSP connection to Microsoft Azure Event Hubs.
3. Create a pipeline that starts with the Microsoft Azure Event Hubs source function. See the Building a pipeline chapter in the Use the manual for instructions on how to build a data pipeline.
4. Configure the Microsoft Azure Event Hubs source function to use your Azure Event Hubs connection. See Get data from Microsoft Azure Event Hubs in the Function Reference manual.
5. (Optional) Convert the byte-encoded data from Azure Event Hubs records into strings that are human-readable during data preview and usable in streaming functions that require string input. See Deserialize and preview data from Microsoft Azure Event Hubs in DSP.

When you activate the pipeline, the source function starts collecting data from Azure Event Hubs. The data is received into the pipeline as a records that contain byte-encoded data values.

Using a dedicated consumer group for each pipeline

To avoid connection failures that can occur when too many receivers try to connect to Azure Event Hubs through the same consumer group, use a dedicated consumer group for each data pipeline that gets data from Azure Event Hubs.

The Connector for Microsoft Azure Event Hubs uses a non-epoch receiver to retrieve data from an event hub through a consumer group. A consumer group can support up to five concurrent non-epoch receivers. If you are using the same consumer group across multiple pipelines, or if you have other programs connecting using this consumer group, then your pipeline can fail to connect. In addition, if an epoch-based receiver connects through the same consumer group, connections for non-epoch based receivers will be rejected.