Configure Pulsar to expose with loadbalancer

In a 1.4.x processing cluster, DSP uses the open-source Pulsar bundle. If you want to configure a topic and use it for ingesting data on the processing cluster, follow the steps below.

1. Enable the proxy.

   dsp config set pulsar proxy_enabled=true

2. Deploy your changes.

   dsp deploy pulsar

3. Validate changes

   kubectl get pods -n pulsar
   <Kubectl get svc -n pulsar

4. Change the service type to a loadbalancer.

   kubectl patch svc pulsar-proxy -n pulsar -p '{"spec": {"type": "LoadBalancer"}}'

5. Patch the node IP as a loadbalancer IP if you do not have a public IP enabled for the cluster.

   curl ifconfig.me
   kubectl patch svc pulsar-proxy -n pulsar -p '{"spec": {"type": "LoadBalancer", "externalIPs":["<IP>"]}}'

6. Create a topic.

   ./pulsar-admin topics create-partitioned-topic persistent://public/default/direct-partitionedmay23 -p 2
   ./pulsar-admin topics list public/default | wc -l 
   ./pulsar-admin namespaces set-retention public/default -s -1 -t 1h

7. Generate and retrieve your certificate.

   kubectl get secret -n pulsar pulsar-tls-broker -o json | jq -r '.data."tls.crt"' | base64 --decode > my-cert.pem
   kubectl get secret -n pulsar pulsar-tls-broker -o json | jq -r '.data."tls.key"' | base64 --decode > my-key.pem 
   kubectl get secret -n pulsar pulsar-tls-broker -o json | jq -r '.data."ca.crt"' | base64 --decode > ca-cert.cert

8. Create an Apache Pulsar connection in your DSP cluster. See Create a DSP connection to Apache Pulsar for more information.
9. Create a pipeline and begin ingesting data.