Splunk® Data Stream Processor

Install and administer the Data Stream Processor

On April 3, 2023, Splunk Data Stream Processor reached its end of sale, and will reach its end of life on February 28, 2025. If you are an existing DSP customer, please reach out to your account team for more information.

All DSP releases prior to DSP 1.4.0 use Gravity, a Kubernetes orchestrator, which has been announced end-of-life. We have replaced Gravity with an alternative component in DSP 1.4.0. Therefore, we will no longer provide support for versions of DSP prior to DSP 1.4.0 after July 1, 2023. We advise all of our customers to upgrade to DSP 1.4.0 in order to continue to receive full product support from Splunk.

About the Splunk App for DSP

The Splunk App for DSP is a monitoring tool that runs on Splunk Enterprise and Splunk Cloud Platform that lets you view the detailed topology and performance information for all of your active pipelines in the Data Stream Processor (DSP).

The Splunk App for DSP comes with several pre-built dashboards to help get you started. These dashboards provide detailed information about the health of your DSP deployment such as:

  • Bytes/MB going out of a source function
  • Bytes/MB going into a sink function
  • Number of active pipelines
  • Memory and CPU usage of the DSP Task and Job Managers
  • Pipeline checkpoint information

Monitor DSP metrics

DSP collects metrics data about your deployment. You can ingest these metrics into your Splunk environment and use Splunk software to analyze the metrics. You can use the pre-built dashboards in the Splunk App for DSP to get a general overview of your DSP deployment as well as visibility into DSP task and job managers, pipelines, ingest metrics, DSP infrastructure and resources, and Pulsar metrics.

Monitor DSP logs

Your DSP deployment generates log files that record detailed messages about events as they happen on your DSP deployment. You can ingest the DSP log files into your Splunk environment and use Splunk software to analyze the log data. The DSP Logs pane in the Splunk App for DSP provides a general overview of the errors and warnings generated in your DSP environment, as well as application log details.

Estimated data volume

The metric and log data volume is dependant upon several variables:

  • The number of errors generated. For example, restarting a pipeline will produce a large number of exceptions.
  • The number of nodes in your DSP cluster.
  • The number of pipelines that are running across all nodes in your DSP cluster.
  • The number of functions used within each pipeline.

Example data volume calculation

Assume that you have a DSP cluster with the following conditions:

  • A default configuration with three nodes in the DSP cluster.
  • Three to six pipelines running at any given moment.
  • Each pipeline has an average of 15 functions.

Under these conditions, you would expect to see around 265 GB of metrics data and 40 GB of log data over a 30 day period.

Last modified on 17 August, 2021
Increase internal partitions to improve pipeline throughput   Install the Splunk App for DSP

This documentation applies to the following versions of Splunk® Data Stream Processor: 1.2.0, 1.2.1-patch02, 1.2.1, 1.2.2-patch02, 1.2.4, 1.2.5, 1.3.0, 1.3.1, 1.4.0, 1.4.1, 1.4.2, 1.4.3, 1.4.4

Was this topic useful?

You must be logged into splunk.com in order to post comments. Log in now.

Please try to keep this discussion focused on the content covered in this documentation topic. If you have a more general question about Splunk functionality or are experiencing a difficulty with Splunk, consider posting a question to Splunkbase Answers.

0 out of 1000 Characters