Using activation checkpoints to activate your pipeline

When a data pipeline is first created, it is in a deactivated state. You must activate a data pipeline to start ingesting data. When a data pipeline is activated for the first time, it starts with no state. Periodically, a checkpoint is added. The checkpoint is used if the data pipeline fails or is restarted. This allows the data pipeline to restart from the point it left off. By default, when a data pipeline is deactivated, the progress state is saved and used the next time the data pipeline is activated. See the Streams API for more information.

Data pipeline activation options

There are three different type of activation options. If you are activating your pipeline for the first time, simply chose Activate in the UI with no other settings enabled.

Activate

In some cases, you may not want to use the latest version of a data pipeline. Set activateLatestVersion = false to use the current version of the data pipeline and ignore the latest version. You can only change this setting using SCloud or the Streams API.

Default: true

Skip Restore State

If you don't want to resume a data pipeline from the latest savepoint, click on Activate Pipeline with Changes from the pipeline canvas view and click on Skip Restore State. Warning: If enabled, your pipeline ignores any data ingested while it is deactivated. If you are activating a pipeline for the first time, you don't need to enable this setting.

Default: false

Allow Non Restored State

In some cases, when you change a data pipeline, the progress state is stored for functions that no longer exist. By default, a data pipeline fails when it is activated while it has an unused progress state. Set allowNonRestoredState = true to allow the data pipeline to ignore any unused progress states, and activate your pipeline without verifying function checkpoints. To enable this setting in the UI, click on Activate Pipeline with Changes from the pipeline canvas view and click on Allow Non Restored State. If skipRestoreState is set to true, this parameter is ignored. If you are activating a pipeline for the first time, you don't need to enable this setting.

Default: false

Setting allowNonRestoredState or skipRestoreState to true can lead to data loss.

Pipeline deactivation options

skipSavepoint

Set skipSavepoint = true to skip saving the data pipeline state when it is deactivated. You can only enable this setting in SCloud or using the Streams API.

Default: false