Increase internal partitions to improve pipeline throughput

The throughput of your pipelines is highly correlated with the parallelism of the pipeline. You can increase the parallelism of certain pipelines by increasing the number of input partitions of the internal Apache Pulsar message bus. The Splunk Data Stream Processor uses Apache Pulsar as the message bus for the following data sources: Read from Splunk Firehose, Read from Forwarders Service, and Read from the Ingest REST API.

Data loss may occur when decreasing the number of partitions later on. Therefore, if you want to increase the number of input partitions, make sure that you do not overallocate input partitions in the process. If you do need to decrease the number of partitions, contact Splunk Support.

Steps:

1. From a controller node in your cluster, get a list of running Apache Pulsar broker pods.

   kubectl get pods -n pulsar

2. Log into a running broker pod.

   kubectl exec -it broker-0 -n pulsar /bin/bash
   

3. (Optional) Get the current number of partitions.

   pulsar-admin topics get-partitioned-topic-metadata DSP/default-ingest/input

4. Use the pulsar-admin CLI tool to update the number of partitions.

   pulsar-admin topics update-partitioned-topic -p <Number-of-Partitions> DSP/default-ingest/input  

5. Confirm that the number of partitions has been changed by using the pulsar-admin CLI tool again.

   pulsar-admin topics get-partitioned-topic-metadata DSP/default-ingest/input
   

6. Log in to the Data Stream Processor and restart your pipelines for changes to take effect.

To further improve pipeline throughput, you can add a batching function in your pipeline. See batch bytes or batch records.