Upgrade the Splunk Data Stream Processor to 1.4.4

This topic describes how to upgrade the Splunk Data Stream Processor (DSP) to 1.4.4.

Upgrading to DSP 1.4.4 requires a new installation of DSP and migrating pipelines from a 1.3.x cluster to the new one. This requires implementation of new infrastructure for your DSP environment due to the simultaneous running of both the old and new clusters. Ensure that you adequately plan for the time and resources that this upgrade requires.

To upgrade to DSP 1.4.4 successfully, you must complete several prerequisite tasks before starting the upgrade. Make sure to read through all of the Before you upgrade sections in this topic and complete the relevant prerequisite tasks before upgrading DSP.

DSP does not provide a means of downgrading to previous versions. If you need to revert to an older DSP release, uninstall the upgraded version and reinstall the version you want.

Before you upgrade

Complete the following tasks before upgrading DSP. If you don't complete these tasks, you might encounter issues such as pipeline failures.

• Make sure you are using a DSP 1.3.x version
• Review known issues
• Review the features planned for deprecation or removal

Make sure you are using a DSP 1.3.x or 1.4.x version

You can upgrade to DSP 1.4.4 from any DSP 1.3.x or 1.4.x version. If you are using DSP 1.2.4, then you must upgrade to a DSP 1.3.x version first. You can upgrade to either DSP 1.3.0 or DSP 1.3.1 from DSP 1.2.4. See Upgrade the Splunk Data Stream Processor from 1.2.4 to 1.3.0 or Upgrade the Splunk Data Stream Processor from 1.3.0 to 1.3.1.

Review known issues

Review the known issues related to the upgrade process. Depending on what functions you have in your pipelines, you might need to complete some additional steps to restore those pipelines after the upgrade is complete.

Review the features planned for deprecation or removal

Review the Features planned for deprecation or removal to see what features are scheduled for future deprecation.

Due to removal of pull-based connectors in DSP 1.4.0, pipelines using pull-based connectors cannot be activated and used in your DSP 1.4.4 cluster.

Install DSP version 1.4.4

See Install the Splunk Data Stream Processor to install DSP version 1.4.4. Review the Installation checklist for DSP to ensure that your environment meets all the requirements for installation. You must install DSP 1.4.4 on a new cluster that can also handle the same data load and configurations as your old cluster. For example, if your 1.3.x cluster has 17 total nodes, then your 1.4.4 cluster should have the same number of nodes with a similar hardware profile.

Back up and restore

Once you've installed DSP 1.4.4 on a new cluster, follow these steps to back up your existing pipelines on your 1.3.x or 1.4.x cluster and restore them in DSP 1.4.4. You do not need to deactivate your pipelines during backup. During the following steps, you will update your data sources to begin sending data to the new cluster and your restored pipelines will begin to read data, while the old cluster will finish reading any remaining data already ingested by the cluster that has yet to be processed.

1. After completing an installation of 1.4.4, complete one of the following steps depending on what DSP version you are upgrading from.
   • If you are upgrading from a 1.4.x version, copy the dsp tool from the bin directory in your 1.4.4 tarball and move it to the working directory of your 1.4.x cluster.
   • If you are upgrading from a 1.3.x version, copy the dsp-gravity tool in the bin directory of your 1.4.4 tarball and move it to the working directory of your 1.3.x cluster.
   You will use either the dsp or dsp-gravity tool to generate a backup in your 1.4.x or 1.3.x cluster and restore that backup in your 1.4.4 cluster.
2. Navigate to a controller node on your DSP 1.3.x or 1.4.x cluster and run one the following commands depending on what version you are upgrading from to generate a backup tarball. For a list of all available flags, see Upgrade flags.
• If you are upgrading from a 1.4.x version, run the following command.

  ./dsp backup --output <name of backup tarball> --target-version=1.4.4

• If you are upgrading from a 1.3.x version, run the following command.

  ./dsp-gravity backup --output <name of backup tarball> --target-version=1.4.4

3. Once your backup is complete, you will be prompted to create and confirm a password to encrypt your backup tarball.

   Please create a password to encrypt the tarball with:
   Enter password: <password>
   Confirm password: <re-entered password>

4. Move your backup tarball from your old cluster to the working directory of your 1.4.4 cluster.
5. Return to your DSP 1.4.4 cluster and run the following command to restore your backup. This restores most cluster configurations and secrets, omitting obsolete configurations or printing out a warning for unexpected configurations. as well as connection secrets, Postgres contents (including pipeline definitions and the DSP license), lookup files, and plugins. For a list of all available flags, see Upgrade flags.

   dsp restore <tarball name>

6. When prompted, enter your password from Step 3 to decrypt your backup tarball and begin restoration in your 1.4.4 cluster.
7. Create new topics or partitions in your external data source to send data into your restored DSP 1.4.4 pipelines.
8. Adjust your restored connections to direct data flow to a new topic or partition in your desired destination.
9. Activate your DSP 1.4.4 pipelines.
10. Configure your external data sources to begin sending data to the new topics or partitions one at a time. Check that the corresponding DSP pipeline correctly receives and processes your data. Continue until you verify that all pipelines are moving data to the correct location and your old cluster does not ingest any new data.

    Both the DSP 1.4.4 and old clusters will actively send data to the same sinks during this step, so ensure that your destination can handle the extra data ingest.

11. Continue to monitor both your DSP 1.4.4 and old clusters. Eventually your old cluster will finish processing its remaining ingested data and can be decommissioned.

After upgrading

After successfully upgrading DSP, complete the following tasks:

• Review known issues and apply workarounds

Review known issues and apply workarounds

There are some known issues that can occur when upgrading. Review the Known issues for DSP topic, and follow any workarounds that apply to you.

Reference

Upgrade flags

The following table lists the main flags you can use with the backup and restore commands and a description of how to use them:

Flag	Description	Associated command
--skip-password	Skips the password encryption of your backup tarball.	dsp backup
--target-version=<version>	Specifies which DSP 1.4.x version is the target of your backup tarball. If you do not provide a version, your target version will default to DSP 1.4.0.	dsp backup
--output=<name of tarball>	Gives the backup TAR file a name. If you do not enter a name, the default name is a standard date-based time.	dsp backup
--cluster-provider=<target cluster provider>	Specifies the target environment where your cluster restores. Cluster provider values include gke or k0s. Only use the gke value if your cluster is based in Google Kubernetes Engine. k0s is the default value if you do not specify a cluster provider.	dsp restore