How to restart your Splunk SOAR (On-premises) cluster

You may need to restart a cluster, or individual cluster node.

Restart using a rolling restart

In most cases, Splunk SOAR (On-premises) clusters should be restarted a node at a time, in what is called a "rolling restart." A rolling restart makes it possible to restart the cluster's nodes without overall cluster downtime, and minimizes impact on ingestion of events and disruption of automation.

Do these steps to do a rolling restart of your Splunk SOAR (On-premises) cluster:

1. Prevent the cluster from routing ingestion and automation actions to the cluster node you want to restart.
   1. Log in to the Splunk SOAR (On-premises) web-based user interface as a user with the administrator role.
   2. From the Home menu, select Administration then Product Settings, then Clustering.
   3. Locate the cluster node you want to restart in the list of nodes. Set the Enabled toggle switch for that node from On to Off.
2. Using SSH, connect to the cluster node you want to restart.
3. From the command line, stop SOAR services on the cluster node.

    <$PHANTOM_HOME>/bin/stop_phantom.sh 

4. From the command line, start SOAR services on the cluster node.

    <$PHANTOM_HOME>/bin/start_phantom.sh 

5. In the web-based user interface, refresh the page at Home menu, select Administration then Product Settings, then Clustering. When the cluster node you just restarted shows as Online proceed to the next step.
6. For the cluster node you just restarted, set the Enabled toggle switch for that node from Off to On. Your cluster can now route ingestion and automation actions to this node.
7. Repeat these steps for each node in your cluster.

Restart a cluster all at once

You may need to restart a cluster all at once. Cluster nodes should be restarted in the reverse order that they were shut down. If you shut down cluster nodes in the order 1, 2, 3, then you should restart those nodes in the order 3, 2, 1.

If you need to restart a cluster all at once, do these steps:

1. Shut down all cluster nodes in order.
   1. Log in to the Splunk SOAR (On-premises) web-based user interface as a user with the administrator role.
   2. From the Home menu, select Administration then Product Settings, then Clustering.
   3. Note the order you disable nodes. For each node, set the Enabled toggle switch for that node from On to Off.
   4. Log out of the Splunk SOAR (On-premises) web-based user interface.
2. Using SSH, connect to the first cluster node, then from the command line, stop SOAR services on the cluster node.

    <$PHANTOM_HOME>/bin/stop_phantom.sh 

3. Repeat for each cluster node.
4. Using SSH, connect to the last cluster node you stopped, then start SOAR services on that node.

    <$PHANTOM_HOME>/bin/start_phantom.sh 

   Repeat this step working your way backward through the list of cluster nodes.
   1. (Conditional) If you do not know the order in which nodes were shutdown, reset RabbitMQ to force a fresh start by using this command on the first node you restart.

       <$PHANTOM_HOME>/bin/phenv rabbitmqctl force_boot

5. Reenable all the cluster nodes in the web-based user interface.
   1. Log in to the Splunk SOAR (On-premises) web-based user interface as a user with the administrator role.
   2. From the Home menu, select Administration then Product Settings, then Clustering.
   3. Reenable each node in the reverse order they were shutdown. For each node, set the Enabled toggle switch for that node from Off to On.