Splunk® Data Stream Processor

Install and administer the Data Stream Processor

Acrobat logo Download manual as PDF

On October 30, 2022, all 1.2.x versions of the Splunk Data Stream Processor will reach its end of support date. See the Splunk Software Support Policy for details. For information about upgrading to a supported version, see the Upgrade the Splunk Data Stream Processor topic.
Acrobat logo Download topic as PDF

About the Splunk App for DSP

The Splunk App for DSP is a monitoring tool that runs on Splunk Enterprise and Splunk Cloud Platform that lets you view the detailed topology and performance information for all of your active pipelines in the Data Stream Processor (DSP).

The Splunk App for DSP comes with several pre-built dashboards to help get you started. These dashboards provide detailed information about the health of your DSP deployment such as:

  • Bytes/MB going out of a source function
  • Bytes/MB going into a sink function
  • Number of active pipelines
  • Memory and CPU usage of the DSP Task and Job Managers
  • Pipeline checkpoint information

Monitor DSP metrics

DSP collects metrics data about your deployment. You can ingest these metrics into your Splunk environment and use Splunk software to analyze the metrics. You can use the pre-built dashboards in the Splunk App for DSP to get a general overview of your DSP deployment as well as visibility into DSP task and job managers, pipelines, ingest metrics, DSP infrastructure and resources, and Pulsar metrics.

Monitor DSP logs

Your DSP deployment generates log files that record detailed messages about events as they happen on your DSP deployment. You can ingest the DSP log files into your Splunk environment and use Splunk software to analyze the log data. The DSP Logs pane in the Splunk App for DSP provides a general overview of the errors and warnings generated in your DSP environment, as well as application log details.

Estimated data volume

The metric and log data volume is dependant upon several variables:

  • The number of errors generated. For example, restarting a pipeline will produce a large number of exceptions.
  • The number of nodes in your DSP cluster.
  • The number of pipelines that are running across all nodes in your DSP cluster.
  • The number of functions used within each pipeline.

Example data volume calculation

Assume that you have a DSP cluster with the following conditions:

  • A default configuration with three nodes in the DSP cluster.
  • Three to six pipelines running at any given moment.
  • Each pipeline has an average of 15 functions.

Under these conditions, you would expect to see around 265 GB of metrics data and 40 GB of log data over a 30 day period.

Last modified on 17 August, 2021
Increase internal partitions to improve pipeline throughput
Install the Splunk App for DSP

This documentation applies to the following versions of Splunk® Data Stream Processor: 1.2.0, 1.2.1-patch02, 1.2.1, 1.2.2-patch02, 1.2.4, 1.2.5, 1.3.0, 1.3.1, 1.4.0, 1.4.1

Was this documentation topic helpful?

You must be logged into splunk.com in order to post comments. Log in now.

Please try to keep this discussion focused on the content covered in this documentation topic. If you have a more general question about Splunk functionality or are experiencing a difficulty with Splunk, consider posting a question to Splunkbase Answers.

0 out of 1000 Characters