Create a DSP connection to Apache Pulsar

To get data from an Apache Pulsar cluster into a data pipeline in Splunk Data Stream Processor, you must first create a connection using the Apache Pulsar Source Connector (SSL Authentication). You can then use the connection in the Apache Pulsar source function to get data from Pulsar into a DSP pipeline.

The Apache Pulsar Connector Using SSL supports two-way SSL authentication where the client and server authenticate each other using the SSL/TLS protocol.

Prerequisites

Before you can create a Pulsar connection, you must have the following:

• An Apache Pulsar cluster version 2.5 or higher, with SSL enabled. Search for "Using TLS with KeyStore configure" in the Apache Pulsar Security manual for more information on enabling SSL in your Apache Pulsar cluster.
• A client private key, a client certificate, and the CA certificate used to sign the client certificate.

Generate the client certificate/key pair

If you don't have a certificate and key pair for the client, signed by a CA trusted by your SSL-enabled Apache Pulsar cluster, then follow the instructions on this page to generate a new client certificate/key pair.

1. From a command-line interface, use openssl to generate a key.

   openssl genrsa -out admin.key.pem 2048

2. Convert the key to PKCS #8 format.

   openssl pkcs8 -topk8 -inform PEM -outform PEM -in admin.key.pem -out admin.key-pk8.pem -nocrypt

3. Generate the certificate request. When you are asked for a common name, enter the role token that you want this key pair to use when authenticating a client.

   openssl req -config openssl.cnf -key admin.key.pem -new -sha256 -out admin.csr.pem

   The OpenSSL configuration file, openssl.cnf, contains the configuration settings and metadata that must be in the certificate request.

4. Sign the request with the certificate authority. The client certificates use the usr_cert extension which allows the certificate to be used for client authentication.

   openssl ca -config openssl.cnf -extensions usr_cert -days 1000 -notext -md sha256 -in admin.csr.pem -out admin.cert.pem

Create the SSL-authenticated Pulsar connection in DSP

Now that you have the required certificates and keys, create a connection in DSP.

1. In DSP, select the Connections page.
2. On the Connections page, click Create Connection.
3. On the Source tab, select Apache Pulsar Source Connector (SSL Authentication) and then click Next.
4. Complete the following fields:

   Field	Description
   Connection Name	A unique name for your connection.
   Description (Optional)	A description of your connection.
   service_url	The SSL-enabled service endpoint for your Pulsar cluster. The prefix of the URL should be pulsar+ssl://
   client_certificate	The file containing the client certificate, beginning with -----BEGIN CERTIFICATE----- and ending with -----END CERTIFICATE-----. This file is created as admin.cert.pem in step 4 of Generate the client certificate/key pair.
   client_private_key	The file containing the client private key, beginning with -----BEGIN RSA PRIVATE KEY----- and ending with -----END RSA PRIVATE KEY-----. This file is created as admin.key-pk8.pem in step 2 of Generate the client certificate/key pair.
   server_ca_certificate (optional)	The file containing the original CA certificate.

   Any credentials that you upload are transmitted securely by HTTPS, encrypted, and securely stored in a secrets manager.

5. Click Save.

   If you're editing a connection that's being used by an active pipeline, you must reactivate that pipeline after making your changes. When you reactivate a pipeline, you must select where you want to resume data ingestion. See Using activation checkpoints to activate your pipeline in the Use the Data Stream Processor manual for more information.

You can now use your connection in the Apache Pulsar source function at the start of your data pipeline to get data from Pulsar. For instructions on how to build a data pipeline, see the Building a pipeline chapter in the Use the Data Stream Processor manual. For information about the source function, see Get data from Apache Pulsar in the Function Reference manual.