Splunk® Data Stream Processor

Connect to Data Sources and Destinations with DSP

On April 3, 2023, Splunk Data Stream Processor reached its end of sale, and will reach its end of life on February 28, 2025. If you are an existing DSP customer, please reach out to your account team for more information.

All DSP releases prior to DSP 1.4.0 use Gravity, a Kubernetes orchestrator, which has been announced end-of-life. We have replaced Gravity with an alternative component in DSP 1.4.0. Therefore, we will no longer provide support for versions of DSP prior to DSP 1.4.0 after July 1, 2023. We advise all of our customers to upgrade to DSP 1.4.0 in order to continue to receive full product support from Splunk.

Create a DSP connection to Apache Pulsar

To get data from an Apache Pulsar cluster into a data pipeline in Splunk Data Stream Processor, you must first create a connection using the Apache Pulsar Source Connector (SSL Authentication). You can then use the connection in the Apache Pulsar source function to get data from Pulsar into a DSP pipeline.

The Apache Pulsar Connector Using SSL supports two-way SSL authentication where the client and server authenticate each other using the SSL/TLS protocol.

Prerequisites

Before you can create a Pulsar connection, you must have the following:

  • An Apache Pulsar cluster version 2.5 or higher, with SSL enabled. Search for "Using TLS with KeyStore configure" in the Apache Pulsar Security manual for more information on enabling SSL in your Apache Pulsar cluster.
  • A client private key, a client certificate, and the CA certificate used to sign the client certificate.

Generate the client certificate/key pair

If you don't have a certificate and key pair for the client, signed by a CA trusted by your SSL-enabled Apache Pulsar cluster, then follow the instructions on this page to generate a new client certificate/key pair.

  1. From a command-line interface, use openssl to generate a key.
    openssl genrsa -out admin.key.pem 2048
  2. Convert the key to PKCS #8 format.
    openssl pkcs8 -topk8 -inform PEM -outform PEM -in admin.key.pem -out admin.key-pk8.pem -nocrypt
  3. Generate the certificate request. When you are asked for a common name, enter the role token that you want this key pair to use when authenticating a client.
    openssl req -config openssl.cnf -key admin.key.pem -new -sha256 -out admin.csr.pem

    The OpenSSL configuration file, openssl.cnf, contains the configuration settings and metadata that must be in the certificate request.

  4. Sign the request with the certificate authority. The client certificates use the usr_cert extension which allows the certificate to be used for client authentication.
    openssl ca -config openssl.cnf -extensions usr_cert -days 1000 -notext -md sha256 -in admin.csr.pem -out admin.cert.pem

Create the SSL-authenticated Pulsar connection in DSP

Now that you have the required certificates and keys, create a connection in DSP.

  1. In DSP, select the Connections page.
  2. On the Connections page, click Create Connection.
  3. On the Source tab, select Apache Pulsar Source Connector (SSL Authentication) and then click Next.
  4. Complete the following fields:
    Field Description
    Connection Name A unique name for your connection.
    Description (Optional) A description of your connection.
    service_url The SSL-enabled service endpoint for your Pulsar cluster. The prefix of the URL should be pulsar+ssl://
    client_certificate The file containing the client certificate, beginning with -----BEGIN CERTIFICATE----- and ending with -----END CERTIFICATE-----. This file is created as admin.cert.pem in step 4 of Generate the client certificate/key pair.
    client_private_key The file containing the client private key, beginning with -----BEGIN RSA PRIVATE KEY----- and ending with -----END RSA PRIVATE KEY-----. This file is created as admin.key-pk8.pem in step 2 of Generate the client certificate/key pair.
    server_ca_certificate (optional) The file containing the original CA certificate.

    Any credentials that you upload are transmitted securely by HTTPS, encrypted, and securely stored in a secrets manager.

  5. Click Save.

    If you're editing a connection that's being used by an active pipeline, you must reactivate that pipeline after making your changes. When you reactivate a pipeline, you must select where you want to resume data ingestion. See Using activation checkpoints to activate your pipeline in the Use the Data Stream Processor manual for more information.

You can now use your connection in the Apache Pulsar source function at the start of your data pipeline to get data from Pulsar. For instructions on how to build a data pipeline, see the Building a pipeline chapter in the Use the Data Stream Processor manual. For information about the source function, see Get data from Apache Pulsar in the Function Reference manual.

Last modified on 25 March, 2022
Connecting Apache Pulsar to your DSP pipeline as a data source   Connecting Google Cloud Pub/Sub to your DSP pipeline

This documentation applies to the following versions of Splunk® Data Stream Processor: 1.3.0, 1.3.1, 1.4.0, 1.4.1, 1.4.2, 1.4.3, 1.4.4, 1.4.5


Was this topic useful?







You must be logged into splunk.com in order to post comments. Log in now.

Please try to keep this discussion focused on the content covered in this documentation topic. If you have a more general question about Splunk functionality or are experiencing a difficulty with Splunk, consider posting a question to Splunkbase Answers.

0 out of 1000 Characters