Monitor your pipeline with data preview and real-time metrics

Use data preview and metrics to investigate, monitor, and troubleshoot your pipeline in real time. You can view metrics on both active and inactive pipelines. Viewing metrics on an active pipeline shows you the rate of all events going in and out of each function.

You can preview data on a per-function basis to see how your data gets transformed in different stages of your pipeline. Data preview also shows the schema of your records and the data type of each field in your record. When previewing data, you can choose how to display the order of output records by dragging the record's fields in the order desired. You can also view your records in table, list, or raw views. These options appear once preview data runs through your pipeline.

To view metrics on an inactive pipeline, start a preview session on your pipeline, allowing sample data to run through your data pipeline. The preview data shows a sample of 100 events, but the preview session will continue collecting metrics until it's stopped or timed out after 15 minutes. You can preview data on full pipelines or on partial pipelines, as long as your partial pipeline includes a data source. Preview data is not available for data sink functions or the union function, and preview data is not actually delivered to any specified destinations.

The start of the sample preview data depends on the initial_position of the data source. If the initial_position is set to "LATEST", then the sample events begin from the latest position of the data stream. If the initial_position is set to "TRIM_HORIZON", then the sample events begin at the very start of the data stream.

Preview data on an inactive pipeline

1. From the Data Pipelines Management page, click on Inactive pipelines and search for the pipeline that you want to view metrics for.
2. Click on the pipeline that you want to view metrics for, and then click Run Preview to allow sample data in through your data pipeline to preview.
3. If you are not already sending data to your pipeline, send data to your pipeline.
4. After a few seconds, live metrics are displayed in each function that shows events per second in and out, bytes per second in and out, and the latency speed that it took to process that event in that function.
5. Metrics get collected every second, and the most recent numbers are shown in the UI.
6. (Optional) Click on each function to see how your data gets sent and transformed at each function in your pipeline. This is helpful for troubleshooting to see where your events may be getting stuck or dropped.

View metrics on an active pipeline

1. From the Data Pipelines Management page, click on Active pipelines and search for the pipeline that you want to view metrics for.
2. Click on the pipeline that you want to view metrics for to be taken to the Read Only canvas view.
3. Live metrics are displayed for each function that shows events per second in and out, bytes per second in and out, and the latency speed that it took to process that event in that function.
4. Metrics get collected every second and the most recent numbers are shown in the UI.

Preview data types

Preview displays the data types of each field in your records. To learn more about DSP data types, see Data Stream Processor Data Types.

Symbol	Data Type
#	number
a	string
B	bytes
	boolean
[]	array
{}	map
(.*)	regex