Add or remove a cluster node from Splunk SOAR (On-premises)

A Splunk SOAR (On-premises) cluster can have nodes added or removed after the cluster has been created.

Splunk SOAR (On-premises) does not have the ability to automatically scale, or automatically add or remove cluster nodes through external systems such as Kubernetes, AWS, or Azure.

Adding cluster nodes

Adding a node to a Splunk SOAR (On-premises) cluster involves building an instance of Splunk SOAR (On-premises) and using the make_cluster_node command on that instance to add it to the cluster.

For more information see these topics in Install and Upgrade Splunk SOAR (On-premises).

• About Splunk SOAR (On-premises) clusters
• Create a Splunk SOAR (On-premises) cluster using an unprivileged installation
• Create a Splunk SOAR (On-premises) cluster in Amazon Web Services
• Run make_cluster_node.pyc

Removing Splunk SOAR (On-premises) cluster nodes

You may want to remove a node from a Splunk SOAR (On-premises) cluster. Possible reasons for removing a cluster node might include; reducing your cluster size, decommissioning or replacing hardware, or even disaster recovery.

Splunk SOAR (On-premises) releases 5.3.0 through 6.0.2 require you to work with Splunk Support to remove cluster nodes from your Splunk SOAR (On-premises) cluster.

Prepare to remove a cluster node

You need to perform several steps before you can safely remove a Splunk SOAR (On-premises) cluster node.

Do these steps to open a support case and install the required tools for removing a cluster node.

1. Open a support case to remove cluster nodes from your Splunk SOAR (On-premises) cluster with Splunk Support.
2. Request the remove_cluster_node.pyc script. Do not proceed until you have the script.
3. Once you have received the remove_cluster_node.pyc script, use SCP to install it on at least one cluster node which will remain in your cluster. Install the script file to <$PHANTOM_HOME>/www/phantom_ui/phadmin/management/commands/remove_cluster_node.pyc
4. Using CHOWN, set the owner of the remove_cluster_node.pyc script the the same user account as the user account that runs Splunk SOAR (On-premises).

   chown <user_account> remove_cluster_node.pyc

5. Verify that the script is correctly installed by getting its help output.

   phenv remove_cluster_node --help

You can safely install the remove_cluster_node.pyc script to all cluster nodes that will stay in your cluster. This may save you time in the future.

Identify the cluster nodes to remove from your cluster

Now that the remove_cluster_node.pyc script is installed, identify the cluster node or nodes you need to remove from your cluster. You can get all the information you need about cluster nodes and clustering components using a management command, phenv cluster_management --status.

Example

phenv cluster_management --status
Splunk SOAR Cluster State:
ClusterNodes found in the database:

  ID: 5c24aeb1-2def-4c3e-b21d-7e44ec8fa9b0
  Name: 10.1.10.1
  Status: Enabled=True Online=True

  ID: 3dce6ace-e11f-4662-baa8-f2de28961abb
  Name: 10.1.10.2
  Status: Enabled=True Online=True

  ID: 128b69b2-99cd-482b-9b60-55cc9419cc1c
  Name: 10.1.10.3
  Status: Enabled=True Online=True

  ID: b2630363-3082-4406-8e39-fc5cdb2d3d03
  Name: 10.1.10.4
  Status: Enabled=True Online=True

  ID: d916c970-525c-41df-97ec-af5700a6d75b
  Name: 10.1.10.5
  Status: Enabled=True Online=True

Consul:

Node                                  Address          Status  Type    Build  Protocol  DC   Segment
3dce6ace-e11f-4662-baa8-f2de28961abb  10.1.10.2:8301   alive   server  1.8.4  2         dc1  <all>
5c24aeb1-2def-4c3e-b21d-7e44ec8fa9b0  10.1.10.1:8301   alive   server  1.8.4  2         dc1  <all>
128b69b2-99cd-482b-9b60-55cc9419cc1c  10.1.10.3:8301   alive   server  1.8.4  2         dc1  <all>
b2630363-3082-4406-8e39-fc5cdb2d3d03  10.1.10.4:8301   alive   client  1.8.4  2         dc1  <all>
d916c970-525c-41df-97ec-af5700a6d75b  10.1.10.5:8301   alive   client  1.8.4  2         dc1  <all>

Consul Leader GUID:

3dce6ace-e11f-4662-baa8-f2de28961abb

Splunk SOAR Leader GUID:

3dce6ace-e11f-4662-baa8-f2de28961abb

RabbitMQ:

Cluster status of node rabbit@10.1.10.2 ...
[{nodes,[{disc,['rabbit@10.1.10.2','rabbit@10.1.10.1','rabbit@10.1.10.3']},{ram,['rabbit@10.1.10.4','rabbit@10.1.10.5']}]},
 {running_nodes,['rabbit@10.1.10.1','rabbit@10.1.10.2','rabbit@10.1.10.3','rabbit@10.1.10.4','rabbit@10.1.10.5']},
 {cluster_name,<<"rabbit@ip-10-1-10-1">>},
 {partitions,[]},
 {alarms,[{'rabbit@10.1.10.1',[]},{'rabbit@10.1.10.2',[]}]},{'rabbit@10.1.10.3',[]}]},{'rabbit@10.1.10.4',[]}]},{'rabbit@10.1.10.5',[]}]}]

Now do these steps:

1. Using SSH, connect to any node in your Splunk SOAR (On-premises) cluster.
2. Get the status of all nodes and components in your cluster.

   phenv cluster_management --status

3. Identify which Splunk SOAR (On-premises) cluster node or nodes are Consul "servers" or "clients".
4. Using the output from the phenv cluster_management --status command, determine the cluster nodes you want to remove from your Splunk SOAR (On-premises) cluster. If you are removing multiple nodes from your cluster at one time, it is best to remove nodes listed as Consul clients before removing nodes listed as Consul servers.

   If a cluster node has already been destroyed, its Consul and RabbitMQ information may not be present in the status output.

5. As a final preparation step, if it has not already been done, set all RabbitMQ nodes to "disc" mode. See The role of RabbitMQ in An overview of the Splunk SOAR (On-premises) clustering feature.

Each node you want to remove must meet the following requirements before being removed from your cluster.

• The node to be removed has already been removed from your load balancer configuration.
• The node to be removed is still listed in the cluster_node table of the Splunk SOAR PostgreSQL database.
• The node to be removed has either:
  • had all Splunk SOAR services permanently stopped or
  • the cluster node has been destroyed

Procedure for removing a Splunk SOAR (On-premises) node

To remove a cluster node follow these steps.

1. Obtain the IP address or the GUID of the cluster node you want to remove from your Splunk SOAR (On-premises) cluster.
2. Prevent the cluster from routing ingestion and automation actions to the cluster node you want to remove. If the cluster node has already been destroyed, skip this step.
   1. Log in to the Splunk SOAR (On-premises) web-based user interface as a user with the administrator role.
   2. From the Home menu, select Administration then Product Settings, then Clustering.
   3. Locate the cluster node you want to remove in the list of nodes. Set the Enabled toggle switch for that node from On to Off. If the cluster node already displays Offline or is already set to Off, skip this step.
3. Using SSH, connect to the cluster node you want to remove. If the cluster node has already been destroyed, skip this step.
4. From the command line, stop SOAR services on the cluster node. If the cluster node has already been destroyed, skip this step.

    <$PHANTOM_HOME>/bin/stop_phantom.sh 

5. Remove the Splunk SOAR (On-premises) node you want to remove from your cluster from your load balancer's configuration. For steps on removing a server from your load balancer's configuration, see the documentation for your load balancer.
6. SSH to a Splunk SOAR (On-premises) cluster node that will remain in your cluster.
7. Run the command to remove the cluster.

   phenv remove_cluster_node <ip_or_guid>

8. Destroy or otherwise deprovision the cluster node that has been removed.

   Splunk SOAR (On-premises) must not be restarted on that deprovisioned cluster node. Restarting Splunk SOAR (On-premises) on the deprovisioned node can interfere with the functioning of the other cluster nodes.

9. Repeat these steps for each cluster node you want to remove from your Splunk SOAR (On-premises) cluster.
10. On all remaining cluster nodes, edit the file <$PHANTOM_HOME>/etc/consul/config.json to remove all references to the removed nodes from the retry_join block.

    This happens automatically for the nodes where phenv remove_cluster_node was run, but must be done manually for each remaining node.

11. Verify cluster membership is as expected.

    phenv cluster_management --status

    Using the management command phenv cluster_management --status will show Consul-related information for recently removed cluster nodes for up to 72 hours after their removal. Consul purges references to those nodes after 72 hours. This is normal and expected.