Splunk® Data Stream Processor

Install and administer the Data Stream Processor

On April 3, 2023, Splunk Data Stream Processor reached its end of sale, and will reach its end of life on February 28, 2025. If you are an existing DSP customer, please reach out to your account team for more information.

All DSP releases prior to DSP 1.4.0 use Gravity, a Kubernetes orchestrator, which has been announced end-of-life. We have replaced Gravity with an alternative component in DSP 1.4.0. Therefore, we will no longer provide support for versions of DSP prior to DSP 1.4.0 after July 1, 2023. We advise all of our customers to upgrade to DSP 1.4.0 in order to continue to receive full product support from Splunk.

Resizing a cluster by adding or removing nodes

You can scale your Kubernetes cluster up or down depending on your organization's needs. There are two main reasons you might want to remove or add a node to the cluster. The first reason is that you want to increase your cluster's capacity for performance and throughput reasons or decrease your cluster's capacity if your resources are being under-utilized. The second reason is if one of your controller nodes goes down and cannot be brought back up. If you are unable to recover a controller node, you must either contact Splunk Support for recovery assistance or reinstall the Data Stream Processor on the irrecoverable node.

The following instructions assume that you have system administrator (root) permissions. If you do not have root permissions, you can use the sudo command.

Removing a node from a cluster

A node can leave a cluster at any time. Any time a node leaves the cluster, pods running on that node are drained and scheduled elsewhere, if possible.

To remove a node from a cluster, do the following steps.

  1. From the working directory of the node that you want to leave the cluster, run:
    dsp leave
  2. (Optional) If the node that you want to remove is in an invalid state, you can force the node to leave your cluster by running the following command from the working directory of a different controller node:
    dsp remove --force <ip-of-failed-node>
  3. This will initiate the leaving process. You can check the progress by running dsp status cluster from a different node's working directory.

You must have at least three nodes minimum in a cluster. If you do not meet this minimum, the cluster will be degraded until you have at least three nodes again.

Adding a node to a cluster

Physical nodes can join a cluster at any time. Any time a new node joins, new capacity becomes available to the cluster for scheduling new pods.

To add a node to the cluster, do the following steps:

  1. From the working directory of an existing cluster node, get the join token.
    dsp print-join 
  2. From the same node, run the below command to join the additional node.
    dsp join-additional -n <number of nodes to join>
  3. On the node that you want to add to the cluster, extract the DSP tarball and navigate to the working directory.
    tar xf <dsp-version>-linux-amd64.tar
    cd <dsp-version>-linux-amd64
  4. Use the join token from step 1 to join this node to the cluster.
    dsp join <existing-node-ip> --port 2222 --public <cert> --private <cert>
  5. (Optional) After the join completes, the new node is ready to accept newly scheduled pods. If you are adding a new node to replace a failed/irrecoverable node, run the following command to check the health of your pods.
     kubectl -n dsp get pods
    If you have pods that are in a PENDING state even after you join a new node, contact Splunk Support for assistance.
Last modified on 13 January, 2023
Data retention policies   Cluster autoscaling for DSP on Google Kubernetes Engine

This documentation applies to the following versions of Splunk® Data Stream Processor: 1.4.0, 1.4.1, 1.4.2, 1.4.3, 1.4.4

Was this topic useful?

You must be logged into splunk.com in order to post comments. Log in now.

Please try to keep this discussion focused on the content covered in this documentation topic. If you have a more general question about Splunk functionality or are experiencing a difficulty with Splunk, consider posting a question to Splunkbase Answers.

0 out of 1000 Characters