About the Splunk App for DSP

The Splunk App for DSP is a monitoring tool that runs on Splunk Enterprise and Splunk Cloud Platform that lets you view the detailed topology and performance information for all of your active pipelines in the Data Stream Processor (DSP).

The Splunk App for DSP comes with several pre-built dashboards to help get you started. These dashboards provide detailed information about the health of your DSP deployment such as:

• Bytes/MB going out of a source function
• Bytes/MB going into a sink function
• Number of active pipelines
• Memory and CPU usage of the DSP Task and Job Managers
• Pipeline checkpoint information

Monitor DSP metrics

DSP collects metrics data about your deployment. You can ingest these metrics into your Splunk environment and use Splunk software to analyze the metrics. You can use the pre-built dashboards in the Splunk App for DSP to get a general overview of your DSP deployment as well as visibility into DSP task and job managers, pipelines, ingest metrics, DSP infrastructure and resources, and Pulsar metrics.

Monitor DSP logs

Your DSP deployment generates log files that record detailed messages about events as they happen on your DSP deployment. You can ingest the DSP log files into your Splunk environment and use Splunk software to analyze the log data. The DSP Logs pane in the Splunk App for DSP provides a general overview of the errors and warnings generated in your DSP environment, as well as application log details.

Estimated data volume

The metric and log data volume is dependant upon several variables:

• The number of errors generated. For example, restarting a pipeline will produce a large number of exceptions.
• The number of nodes in your DSP cluster.
• The number of pipelines that are running across all nodes in your DSP cluster.
• The number of functions used within each pipeline.

Example data volume calculation

Assume that you have a DSP cluster with the following conditions:

• A default configuration with three nodes in the DSP cluster.
• Three to six pipelines running at any given moment.
• Each pipeline has an average of 15 functions.

Under these conditions, you would expect to see around 265 GB of metrics data and 40 GB of log data over a 30 day period.