Recreate warm standby after a failover

After a failover, the previous warm standby is now a standalone primary instance of and the previous primary is offline or otherwise unavailable. A administrator can reconfigure these two instances into a new warm standby pair.

For the rest of this topic the two instances will be referred to as either instance A or instance B.

Instance A

The original primary instance.

Instance B

The original warm standby instance of .

Configure instance B as the primary and instance A as the warm standby

This is the easiest way to reconfigure the instances for warm standby after a failover.

The initial states for your instances must be:

• Instance A, the original primary is online but services are not running.
• Instance B, the former warm standby is now a stand alone instance.

If the Splunk Phantom instances are not in these states, stop. Evaluate if another option is more appropriate for your needs.

Do these steps as either the root user or a user with sudo permissions.

1. SSH to instance A.
   SSH <username>@<instance_A_hostname>
   1. Start PostgreSQL.
      /<PHANTOM_HOME>/bin/phsvc start postgresql-11
   2. Start pgbouncer.
      /<PHANTOM_HOME>/bin/phsvc start pgbouncer
   3. Turn off primary mode
      phenv python /<PHANTOM_HOME>/bin/setup_warm_standby.pyc --primary-mode --off
2. SSH to instance B.
   SSH <username>@<instance_B_hostname>
   1. Configure instance B as the new primary.
      phenv python /<PHANTOM_HOME>/bin/setup_warm_standby.pyc --primary-mode --configure --primary-ip <primary_ip> --standby-ip <standby_ip>
3. SSH to instance A.
   SSH <username>@<instance_A_hostname>
   1. Configure instance A as the new warm standby.
      phenv python /<PHANTOM_HOME>/bin/setup_warm_standby.pyc --standby-mode --configure --primary-ip <primary_ip> --standby-ip <standby_ip>

Now the two instances are configured for warm standby. Instance B is now the primary and instance A is now the warm standby.

Configure instance A as the primary and instance B as the warm standby

This option returns the instances to the same roles they served before the failover. This can be done after you have configured the instance B as the primary using the steps in the earlier section.

Each time warm standby is configured the database on the standby instance is erased and the entire Splunk Phantom PostgreSQL database has to be streamed from the primary.

The initial states for your instances must be:

• Instance B, the former warm standby is now a stand alone instance. All services are running.
• Instance A, the original primary is configured as the warm standby. All services are running.

If the Splunk Phantom instances are not in these states, stop. Evaluate if another option is more appropriate for your needs.

1. SSH to instance B.
   SSH <username>@<instance_B_hostname>
   1. Stop services.
      /<PHANTOM_HOME>/bin/stop_phantom.sh
2. SSH to instance A.
   SSH <username>@<instance_A_hostname>
   1. Configure instance A as the primary.
      phenv python /<PHANTOM_HOME>/bin/setup_warm_standby.pyc --standby-mode --convert-to-primary
      Warm standby is disabled. Instance A is a standalone instance, while instance B is idle and all services have been shut down.

      If the Splunk Phantom instances are not in the described states, stop. Check for and do any steps which have been missed before proceeding.

3. SSH to instance B.
   SSH <username>@<instance_B_hostname>
   1. Start PostgreSQL.
      /<PHANTOM_HOME>/bin/phsvc start postgresql-11
   2. Start pgbouncer.
      /<PHANTOM_HOME>/bin/phsvc start pgbouncer
   3. Turn off primary mode
      phenv python /<PHANTOM_HOME>/bin/setup_warm_standby.pyc --primary-mode --off
4. SSH to instance A.
   SSH <username>@<instance_A_hostname>
   1. Configure instance A as primary.
      phenv python /<PHANTOM_HOME>/bin/setup_warm_standby.pyc --primary-mode --configure --primary-ip <primary_ip> --standby-ip <standby_ip>
5. SSH to instance B.
   SSH <username>@<instance_B_hostname>
   1. Configure instance B as the warm standby.
      phenv python /<PHANTOM_HOME>/bin/setup_warm_standby.pyc --standby-mode --configure --primary-ip <primary_ip> --standby-ip <standby_ip>

Instance A and B are configured as a warm standby pair. Instance A is the primary, and instance B is the warm standby.