Resizing a cluster by adding or removing nodes

You can scale your Kubernetes cluster up or down depending on your organization's needs. There are two main reasons you might want to remove or add a node to the cluster. The first reason is that you want to increase your cluster's capacity for performance and throughput reasons or decrease your cluster's capacity if your resources are being under-utilized. The second reason is if one of your master nodes goes down and cannot be brought back up. If you are unable to recover a master node, you must either contact Splunk Support for recovery assistance or reinstall the Data Stream Processor on the irrecoverable node.

The following instructions assume that you have system administrator (root) permissions. If you do not have root permissions, you can use the sudo command.

Removing a node from a cluster

A node can leave a cluster at any time. Any time a node leaves the cluster, pods running on that node are drained and scheduled elsewhere, if possible.

To remove a node from a cluster, do the following steps.

1. From the working directory of the node that you want to leave the cluster, run:

   ./leave

2. (Optional) If the node that you want to remove is in an invalid state, you can force the node to leave your cluster by running the following command from the working directory of a different master node:

   gravity remove --force <ip-of-failed-node>

3. This will initiate the leaving process. You can check the progress by running gravity status from a different node's working directory.

You must have at least three nodes minimum in a cluster. If you do not meet this minimum, the cluster will be degraded until you have at least three nodes again.

Adding a node to a cluster

Physical nodes can join a cluster at any time. Any time a new node joins, new capacity becomes available to the cluster for scheduling new pods.

To add a node to the cluster, do the following steps:

1. From the working directory of an existing cluster node, get the join token.

   gravity status --token 

2. On the node that you want to add to the cluster, extract the DSP tarball and navigate to the working directory.

   tar xf <dsp-version>-linux-amd64.tar
   cd <dsp-version>-linux-amd64
   

3. Use the join token from step 1 to join this node to the cluster.

   ./join <existing-node-ip> --token=<token> --role=worker

4. (Optional) After the join completes, the new node is ready to accept newly scheduled pods. If you are adding a new node to replace a failed/irrecoverable node, run the following command to check the health of your pods.

    kubectl -n dsp get pods

   If you have pods that are in a PENDING state even after you join a new node, contact Splunk Support for assistance.