Deserialize and send Azure Event Hubs data from a DSP pipeline

Your pipeline can ingest data from Azure Event Hubs and send that data to a Splunk Index or a third-party sink.

Prerequisites

• A connection to read from Azure Event Hub's namespace. See the Azure Event Hub Connector in the Getting data in manual.
• A standalone Event Hub consumer group per pipeline, see Create or update consumer groups.

This connector uses a non-epoch receiver to retrieve data from an Azure Event Hubs through a consumer group. An Event Hub consumer group can support up to five concurrent non-epoch receivers. If you are using the same consumer group across multiple pipelines, or if you have other programs connecting using this consumer group, then your pipeline can fail to connect. In addition, if an epoch-based receiver connects through the same consumer group, connections for non-epoch based receivers will be rejected. To avoid the risk of having too many receivers connecting through the same consumer group, or other programs connecting with epoch-based receivers, best practices are to create a dedicated consumer group for each pipeline.

Create a pipeline

Once you satisfy the previous prerequisites, you can ingest data from Azure Event Hubs.

1. From the Add Data page, select Read from Azure Event Hubs using SAS Key as your source function.
2. On the next page, complete the following fields:

   Field	Description	Example
   Connection Id	The name of the connection set up as a prerequisite.	myEventHubConnection
   Event Hub name	The name of the Event Hub entity to subscribe to.	operational-logs
   Consumer Group Name	The name of the consumer group set up as a prerequisite.	dsp-ConsumerGroup
   Starting Position	Optional. Specifies where to start reading data from. Defaults to earliest.	earliest

3. Click the + to add a new function.
4. Select Eval.
5. Azure Event Hubs transmit data in binary and does not follow the DSP event schema. You must convert your data from bytes. Use the Eval function's textbox to call deserialize_json_object to turn your bytes into a map.

   body = deserialize_json_object(body)
   

6. Click the + icon and add any other data transformation functions to your pipeline. See the function reference for a full list of available functions.
7. Click the + icon to add a new function to your pipeline.
8. Select the Write to the Splunk platform with Batching function.
9. Configure the Write to the Splunk platform with Batching function. See Write to the Splunk platform with batching for more detailed information on these fields.

   Field	Example
   index	null
   default_index	"main"
   parameters	Optional. For a list of available parameters, see the Write to the Splunk platform with Batching function reference.

10. Click Validate to confirm your pipeline's functions are correctly configured.
11. Click Save to save your pipeline, or Activate to activate it.