Connecting Google Cloud Pub/Sub to your DSP pipeline

When creating a data pipeline in the , you can connect to Google Cloud Pub/Sub and use it as a data source. You can get message data from Pub/Sub topics into a pipeline, transform the data as needed, and then send the transformed data out from the pipeline to a destination of your choosing.

To connect to Google Cloud Pub/Sub as a data source, you must complete the following tasks:

1. Create a connection that allows the DSP to access your Pub/Sub data. See Create a DSP connection to Google Cloud Pub/Sub.
2. Create a pipeline that starts with the Google Cloud Pub/Sub source function. See the Building a pipeline chapter in the Use the Data Stream Processor manual for instructions on how to build a data pipeline.
3. Configure the Google Cloud Pub/Sub source function to use your Pub/Sub connection and receive data from a Pub/Sub topic. See Get data from Google Cloud Pub/Sub in the Function Reference manual.
4. (Optional) Convert the body field in the Pub/Sub records from bytes to a more commonly supported data type such as string. This conversion makes the field human-readable during data preview and compatible with a wider range of streaming functions. See Deserialize and preview data from Google Cloud Pub/Sub in the .

When you activate the pipeline, the source function subscribes to the specified Pub/Sub topic and starts receiving any messages that are delivered to the topic after the subscription was created. Each message is received into the pipeline as a record containing byte-encoded data.

If your data fails to get into the pipeline, check the connection settings to make sure you have the correct credentials file for your Google Cloud service account.