Recreate warm standby after a failover

After a failover, the previous warm standby is now a standalone primary instance of Splunk Phantom and the previous primary is offline or otherwise unavailable. A Splunk Phantom administrator can reconfigure these two instances into a new warm standby pair.

For the rest of this topic the two Splunk Phantom instances will be referred to as either instance A or instance B.

Instance A

The original primary Splunk Phantom instance.

Instance B

The original warm standby instance of Splunk Phantom.

Configure instance B as the primary and instance A as the warm standby

This is the easiest way to reconfigure the instances for warm standby after a failover.

The initial states for your instances must be:

• Instance A, the original primary is online but Splunk Phantom services are not running.
• Instance B, the former warm standby is now a stand alone Splunk Phantom instance.

If the Splunk Phantom instances are not in these states, stop. Evaluate if another option is more appropriate for your needs.

Do these steps as either the root user or a user with sudo permissions.

1. SSH to instance A.
   SSH <username>@<instance_A_hostname>
   1. Start PostgreSQL.
      /<PHANTOM_HOME>/bin/phsvc start postgresql-9.4
   2. Start pgbouncer.
      /<PHANTOM_HOME>/bin/phsvc start pgbouncer
   3. Turn off primary mode
      phenv python2.7 /<PHANTOM_HOME>/bin/setup_warm_standby.pyc --primary-mode --off
2. SSH to instance B.
   SSH <username>@<instance_B_hostname>
   1. Configure instance B as the new primary.
      phenv python2.7 /<PHANTOM_HOME>/bin/setup_warm_standby.pyc --primary-mode --configure --primary-ip <primary_ip> --standby-ip <standby_ip>
3. SSH to instance A.
   SSH <username>@<instance_A_hostname>
   1. Configure instance A as the new warm standby.
      phenv python2.7 /<PHANTOM_HOME>/bin/setup_warm_standby.pyc --standby-mode --configure --primary-ip <primary_ip> --standby-ip <standby_ip>

Now the two instances are configured for warm standby. Instance B is now the primary and instance A is now the warm standby.

Configure instance A as the primary and instance B as the warm standby

This option returns the instances to the same roles they served before the failover. This can be done after you have configured the instance B as the primary using the steps in the earlier section.

Each time warm standby is configured the database on the standby instance is erased and the entire Splunk Phantom PostgreSQL database has to be streamed from the primary.

The initial states for your instances must be:

• Instance B, the former warm standby is now a stand alone Splunk Phantom instance. All Splunk Phantom services are running.
• Instance A, the original primary is configured as the warm standby. All Splunk Phantom services are running.

If the Splunk Phantom instances are not in these states, stop. Evaluate if another option is more appropriate for your needs.

1. SSH to instance B.
   SSH <username>@<instance_B_hostname>
   1. Stop Splunk Phantom services.
      /<PHANTOM_HOME>/bin/stop_phantom.sh
2. SSH to instance A.
   SSH <username>@<instance_A_hostname>
   1. Configure instance A as the primary.
      phenv python2.7 /<PHANTOM_HOME>/bin/setup_warm_standby.pyc --standby-mode --convert-to-primary
      Warm standby is disabled. Instance A is a standalone Splunk Phantom instance, while instance B is idle and all Splunk Phantom services have been shut down.

      If the Splunk Phantom instances are not in the described states, stop. Check for and do any steps which have been missed before proceeding.

3. SSH to instance B.
   SSH <username>@<instance_B_hostname>
   1. Start PostgreSQL.
      /<PHANTOM_HOME>/bin/phsvc start postgresql-9.4
   2. Start pgbouncer.
      /<PHANTOM_HOME>/bin/phsvc start pgbouncer
   3. Turn off primary mode
      phenv python2.7 /<PHANTOM_HOME>/bin/setup_warm_standby.pyc --primary-mode --off
4. SSH to instance A.
   SSH <username>@<instance_A_hostname>
   1. Configure instance A as primary.
      phenv python2.7 /<PHANTOM_HOME>/bin/setup_warm_standby.pyc --primary-mode --configure --primary-ip <primary_ip> --standby-ip <standby_ip>
5. SSH to instance B.
   SSH <username>@<instance_B_hostname>
   1. Configure instance B as the warm standby.
      phenv python2.7 /<PHANTOM_HOME>/bin/setup_warm_standby.pyc --standby-mode --configure --primary-ip <primary_ip> --standby-ip <standby_ip>

Instance A and B are configured as a warm standby pair. Instance A is the primary, and instance B is the warm standby.