Recreate warm standby after a failover

After a failover, the previous warm standby is now a standalone primary instance of and the previous primary is offline or otherwise unavailable. A administrator can reconfigure these two instances into a new warm standby pair.

For the rest of this topic the two instances will be referred to as either instance A or instance B.

Instance A

The original primary instance.

Instance B

The original warm standby instance of .

Configure instance B as the primary and instance A as the warm standby

This is the easiest way to reconfigure the instances for warm standby after a failover.

The initial states for your instances must be:

• Instance A, the original primary is online but services are not running.
• Instance B, the former warm standby is now a stand alone instance.

If the Splunk SOAR (On-premises) instances are not in these states, stop. Evaluate if another option is more appropriate for your needs.

Do these steps as the phantom user.

1. SSH to instance A.
   SSH <username>@<instance_A_hostname>
   1. Start PostgreSQL.
      /<PHANTOM_HOME>/bin/phsvc start postgresql-11
   2. Start pgbouncer.
      /<PHANTOM_HOME>/bin/phsvc start pgbouncer
   3. Turn off primary mode
      phenv python /<PHANTOM_HOME>/bin/setup_warm_standby.pyc --primary-mode --off
2. SSH to instance B.
   SSH <username>@<instance_B_hostname>
   1. Configure instance B as the new primary.
      phenv python /<PHANTOM_HOME>/bin/setup_warm_standby.pyc --primary-mode --configure --primary-ip <primary_ip> --standby-ip <standby_ip>
3. SSH to instance A.
   SSH <username>@<instance_A_hostname>
   1. Configure instance A as the new warm standby.
      phenv python /<PHANTOM_HOME>/bin/setup_warm_standby.pyc --standby-mode --configure --primary-ip <primary_ip> --standby-ip <standby_ip>

Now the two instances are configured for warm standby. Instance B is now the primary and instance A is now the warm standby.

Configure instance A as the primary and instance B as the warm standby

This option returns the instances to the same roles they served before the failover. This can be done after you have configured the instance B as the primary using the steps in the earlier section.

Each time warm standby is configured the database on the standby instance is erased and the entire Splunk SOAR (On-premises) PostgreSQL database has to be streamed from the primary.

The initial states for your instances must be:

• Instance B, the former warm standby is now a stand alone instance. All services are running.
• Instance A, the original primary is configured as the warm standby. All services are running.

If the Splunk SOAR (On-premises) instances are not in these states, stop. Evaluate if another option is more appropriate for your needs.

1. SSH to instance B.
   SSH <username>@<instance_B_hostname>
   1. Stop services.
      /<PHANTOM_HOME>/bin/stop_phantom.sh
2. SSH to instance A.
   SSH <username>@<instance_A_hostname>
   1. Configure instance A as the primary.
      phenv python /<PHANTOM_HOME>/bin/setup_warm_standby.pyc --standby-mode --convert-to-primary
      Warm standby is disabled. Instance A is a standalone instance, while instance B is idle and all services have been shut down.

      If the Splunk SOAR (On-premises) instances are not in the described states, stop. Check for and do any steps which have been missed before proceeding.

3. SSH to instance B.
   SSH <username>@<instance_B_hostname>
   1. Start PostgreSQL.
      /<PHANTOM_HOME>/bin/phsvc start postgresql-11
   2. Start pgbouncer.
      /<PHANTOM_HOME>/bin/phsvc start pgbouncer
   3. Turn off primary mode
      phenv python /<PHANTOM_HOME>/bin/setup_warm_standby.pyc --primary-mode --off
4. SSH to instance A.
   SSH <username>@<instance_A_hostname>
   1. Configure instance A as primary.
      phenv python /<PHANTOM_HOME>/bin/setup_warm_standby.pyc --primary-mode --configure --primary-ip <primary_ip> --standby-ip <standby_ip>
5. SSH to instance B.
   SSH <username>@<instance_B_hostname>
   1. Configure instance B as the warm standby.
      phenv python /<PHANTOM_HOME>/bin/setup_warm_standby.pyc --standby-mode --configure --primary-ip <primary_ip> --standby-ip <standby_ip>

Instance A and B are configured as a warm standby pair. Instance A is the primary, and instance B is the warm standby.