Splunk® Data Stream Processor

Install and administer the Data Stream Processor

On April 3, 2023, Splunk Data Stream Processor reached its end of sale, and will reach its end of life on February 28, 2025. If you are an existing DSP customer, please reach out to your account team for more information.

All DSP releases prior to DSP 1.4.0 use Gravity, a Kubernetes orchestrator, which has been announced end-of-life. We have replaced Gravity with an alternative component in DSP 1.4.0. Therefore, we will no longer provide support for versions of DSP prior to DSP 1.4.0 after July 1, 2023. We advise all of our customers to upgrade to DSP 1.4.0 in order to continue to receive full product support from Splunk.

Increase internal partitions to improve pipeline throughput

The throughput of your pipelines is highly correlated with the parallelism of the pipeline. You can increase the parallelism of certain pipelines by increasing the number of input partitions of the internal Apache Pulsar message bus. The Splunk Data Stream Processor uses Apache Pulsar as the message bus for the following data sources: Read from Splunk Firehose, Read from Forwarders Service, and Read from the Ingest REST API.

Data loss may occur when decreasing the number of partitions later on. Therefore, if you want to increase the number of input partitions, make sure that you do not overallocate input partitions in the process. If you do need to decrease the number of partitions, contact Splunk Support.


  1. From a controller node in your cluster, get a list of running Apache Pulsar broker pods.
    kubectl get pods -n pulsar
  2. Log into a running broker pod.
    kubectl exec -it pulsar-broker-0 -n pulsar /bin/bash
  3. (Optional) Get the current number of partitions.
    pulsar-admin topics get-partitioned-topic-metadata persistent://dsp/ingest_default/input
  4. Use the pulsar-admin CLI tool to update the number of partitions.
    pulsar-admin topics update-partitioned-topic -p <Number-of-Partitions> persistent://dsp/ingest_default/input  
  5. Confirm that the number of partitions has been changed by using the pulsar-admin CLI tool again.
    pulsar-admin topics get-partitioned-topic-metadata persistent://dsp/ingest_default/input
  6. Log in to the Data Stream Processor and restart your pipelines for changes to take effect.

To further improve pipeline throughput, you can add a batching function in your pipeline. See batch bytes or batch records.

Last modified on 14 November, 2023
Back up your Splunk Data Stream Processor deployment   About the Splunk App for DSP

This documentation applies to the following versions of Splunk® Data Stream Processor: 1.4.0, 1.4.1, 1.4.2, 1.4.3, 1.4.4

Was this topic useful?

You must be logged into splunk.com in order to post comments. Log in now.

Please try to keep this discussion focused on the content covered in this documentation topic. If you have a more general question about Splunk functionality or are experiencing a difficulty with Splunk, consider posting a question to Splunkbase Answers.

0 out of 1000 Characters