Modify Ingest Processor pipelines

You can view, edit, and, apply, remove, and delete pipelines from the Pipelines page in your tenant. Select a pipeline to view information, such as the description of the pipeline, the number of applied pipelines, and the SPL2 statement that it is using.

For instructions on how to work with pipelines, see the following sections:

• Apply a pipeline
• Remove a pipeline
• Edit a pipeline
• Refresh a pipeline
• Delete a pipeline

Apply a pipeline

To begin data ingestion, apply a pipeline in your Ingest Processor.

1. Navigate to the Pipelines page.
2. In the row that lists the pipeline you want to apply, select the Actions icon () and then select Apply.
3. Confirm that you want to apply your pipeline by selecting Apply in the pop-up window. It can take a few minutes for the Ingest Processor service to finish applying your pipeline. During this time, all applied pipelines enter the Pending status. Once the operation is complete, the Pending Apply status icon () stops displaying beside the pipeline, and all affected pipelines transition from the Pending status back to the Healthy status.
4. (Optional) Refresh your browser to check if the Pending Apply status icon () is no longer displaying.

The applied pipeline can now process and route data according to the instructions specified in your SPL2 statement.

If you apply and remove the same pipeline in quick succession, then the status icons might present conflicting information until one of the processes is completed.

Remove a pipeline

Ingest Processor acts on the data that it receives based on the data processing instructions defined in the associated pipelines. To disable specific data processing operations on your Ingest Processor, remove the pipeline that defines those instructions.

Removing a pipeline does not delete it from your Pipelines page, it stops the active data ingestion. See Delete a pipeline for more information on deleting a pipeline.

Before removing a pipeline from Ingest Processor, check the other pipelines that are applied to the Ingest Processor and verify that removing the pipeline won't cause unexpected data loss. Any data that is not processed by a pipeline is treated as unprocessed data, which is either routed to the default destination or dropped. As a best practice for preventing unwanted data loss, make sure to always have a default destination for your Ingest Processor. See Data pathway for more information.

1. Navigate to the Pipelines page.
2. In the row that lists the pipeline you want to apply, select the Actions icon () and then select Remove.
3. Confirm that you want to apply your pipeline by selecting Remove in the pop-up window. It can take a few minutes for the Ingest Processor service to finish removing your pipeline. During this time, all applied pipelines enter the Pending status. Once the operation is complete, the Pending Removal status icon () stops displaying beside the pipeline, and all affected pipelines transition from the Pending status back to the Healthy status.
4. (Optional) Refresh your browser to check if the Pending Removal status icon () is no longer displaying.

The removed pipeline will no longer process and route data according to the instructions specified in the pipeline.

If you apply and remove the same pipeline from your Ingest Processor in quick succession, then the status icons might present conflicting information until one of the processes is completed.

Edit a pipeline

You can modify the SPL2 statement of a pipeline or select a different data source or destination for the pipeline by completing the following steps.

1. Navigate to the Pipelines page.
2. In the row that lists your pipeline, select the Actions icon () and select Edit.
3. Modify the pipeline as needed, and then select Save pipeline.
4. Verify that your pipeline edits are as desired, and then click Save to continue.

If the pipeline was applied to your Ingest Processor before you edited it, then the Ingest Processor re-applies the edited pipeline. It can take a few minutes for the Ingest Processor to finish applying your pipeline. During this time, the applied pipeline enters the Pending status. Once the operation is complete, the Pending Apply status icon () stops displaying beside the pipeline, and the Ingest Processor transitions from the Pending status back to the Healthy status. Refresh your browser to check if the icon is no longer displaying.

Refresh a pipeline

When your pipeline is applied, the pipeline does not automatically detect configuration changes to the source type or the destination that it is using. To refresh a pipeline so that it uses the latest source type and destination configurations, do either of the following:

• Edit the pipeline and save your changes. You must edit the pipeline in a way that changes how it works. For example, adding a line of empty space after the SPL2 statement does not cause the pipeline to be refreshed, but adding a processing command causes a refresh.
• Remove the pipeline and then re-apply it.

Be aware that removing and re-applying a pipeline can create some gaps in your data. If your data sources send data during that time, that data doesn't get processed by the pipeline that you're re-applying.

Delete a pipeline

You can delete a pipeline from the system by completing the following steps.

Prerequisites

The pipeline must not be applied. See Remove a pipeline for more information on removing a pipeline to stop its data ingestion.

Steps

1. Navigate to the Pipelines page.
2. In the row that lists your pipeline, select the Actions icon () and select Delete.
3. When prompted to confirm your choice, select Delete.

The Ingest Processor service returns a message confirming that your pipeline was deleted successfully.