All DSP releases prior to DSP 1.4.0 use Gravity, a Kubernetes orchestrator, which has been announced end-of-life. We have replaced Gravity with an alternative component in DSP 1.4.0. Therefore, we will no longer provide support for versions of DSP prior to DSP 1.4.0 after July 1, 2023. We advise all of our customers to upgrade to DSP 1.4.0 before July 1, 2023 in order to continue to receive full product support from Splunk.
Using activation checkpoints to activate your pipeline
When you first create a pipeline, it is in a deactivated state. You must activate a pipeline to start ingesting data. The first time you activate a pipeline, it starts with no state. The frequently adds checkpoints to active pipelines which are used if the pipeline fails or is restarted. By default, when a data pipeline is deactivated, the progress state is saved as a savepoint and used the next time the data pipeline is activated.
Checkpoints and savepoints
Checkpoints and savepoints are two separate features in the that serve different needs to ensure consistency and fault tolerance, and to make sure that no data is dropped when you deactivate and reactivate a pipeline. Checkpoints are primarily used to provide a recovery mechanism in case of unexpected errors. The adds checkpoints frequently and automatically, without any user interaction. In contrast, savepoints are created on a best-effort basis after a particular user action, such as deactivating a pipeline. Savepoints ensure that your pipeline picks up right where it left off when you reactivate it by saving the progress state of each function in the pipeline.
Data pipeline activation options
There are three different types of activation options. If you are activating your pipeline for the first time, simply chose Activate in the UI with no other settings enabled. See the table below for information on the different activation options available.
|Activate a pipeline. Use this option first before using the other activation options.
|If you are reactivating a previously activated pipeline, this option reactivates the pipeline from the pipeline savepoint.
|Skip Restore State
|Reactivate a pipeline without attempting to restore any state from a savepoint or, in the case that a savepoint doesn't exist, a checkpoint.
|When you use this option, the pipeline starts ingesting data from the initial position of your data source, ignoring the last savepoint. If the initial position of your data source is earliest, or
TRIM HORIZON, then you could experience either data duplication for data still in the stream that was already ingested or data loss for data that may have expired in the stream before it could be ingested. If the initial position of your data source is
LATEST, then you may experience data loss, since data sent after the last checkpoint was saved and the latest data in the input stream would be missed.
|Allow Non Restored State
|Reactivate a pipeline using a best effort, partial restore from the savepoint.
|Modifying a pipeline might prevent the existing savepoint from being restored which normally results in a pipeline activation failure. When you use this option, the will use the existing savepoint to recover as much state as possible and ignore all errors while doing so. Use this option instead of Skip Restore State when you want to use the last savepoint to recover as much state as possible.
Enabling Allow Non Restored State or Skip Restore State can lead to data loss.
Pipeline deactivation options
In addition to pipeline activation options, you also have the following deactivation option available.
|Deactivate a pipeline without creating a savepoint.
|By default, the creates a savepoint when you deactivate a pipeline to ensure that your pipeline picks up right where it left off when you reactivate it. However, attempts to deactivate your pipeline might fail if the pipeline is blocked and no data is flowing through, which prevents a savepoint from being created. To bypass this issue, enable this setting to deactivate the pipeline without creating a savepoint. In most cases, you will not experience data loss when you reactivate the pipeline as the will use the last checkpoint instead. However, if the pipeline was blocked for a long time, data loss might occur as an older checkpoint will be used. You can only modify this setting using the Splunk Cloud Services CLI or the Streams service.
Back up, restore, and share pipelines
Interpreting pipeline statuses
This documentation applies to the following versions of Splunk® Data Stream Processor: 1.2.0, 1.2.1-patch02, 1.2.1, 1.2.2-patch02, 1.2.4, 1.2.5, 1.3.0, 1.3.1, 1.4.0, 1.4.1, 1.4.2, 1.4.3