Use ibackup.pyc with warm standby

The warm standby and the backup and restore features require careful planning to use together.

Warm standby and ibackup features of Splunk Phantom use the Write Ahead Logging feature in PostgreSQL. When you configure a Splunk Phantom deployment to use both warm standby and ibackup, you must configure warm standby first. After restoring a deployment with ibackup, you must update the warm standby configuration.

Configuring warm standby after configuring ibackup archives all existing backups. Archiving all of the backups prevents new backups from being generated or existing backups from being used in a restore.

You can generate new backups once you run ibackup with the --setup option.

Restore a system configured for warm standby

In a warm standby configuration, when the primary Phantom instance is restored from a backup, you must update the warm standby configuration.

Prerequisites

You need the following information to update your warm standby configuration:

• Password for the Splunk Phantom user on the secondary Phantom instance. If the Splunk Phantom user does not have a password, you must set one.
• Password for the PostgreSQL database replication user.
• Configuration information for creating the SSL certificate:
  • Country code
  • State code
  • Organization
  • Organization unit
  • Domain
  • Email

Restore a backup from a warm standby primary to the same Splunk Phantom instance

When you restore a backup of a Phantom warm standby primary to the same instance, the warm standby configuration must be updated.

To update the warm standby configuration, perform the following steps:

1. Open a terminal session for both the primary and secondary Splunk Phantom instances. Keep these sessions open until you complete these steps.
   1. From the command line, SSH to your primary Splunk Phantom instance.
      
      SSH <username>@<primary_phantom_hostname>
   2. SSH to your secondary and warm standby Splunk Phantom instance.
      
      SSH <username>@<warm_standby_phantom_hostname>
   3. In both sessions, elevate to root.
      
      sudo su -
2. On the primary instance of Splunk Phantom, perform the restore. See Restore Splunk Phantom from a backup.
3. On the primary instance of Splunk Phantom, disable warm standby.
   
   phenv python /<PHANTOM_HOME>/bin/setup_warm_standby.pyc --primary-mode --off
4. On the secondary instance of Splunk Phantom, disable warm standby.
   
   phenv python /<PHANTOM_HOME>/bin/setup_warm_standby.pyc --standby-mode --off
5. On the secondary instance of Splunk Phantom, stop all Splunk Phantom services.
   
   /<PHANTOM_HOME>/bin/stop_phantom.sh
   
   

   Failing to stop Phantom services on the secondary instance results in two active Splunk Phantom instances operating independently, polling for data and executing automated actions. This can result in data loss or other undesired results.

6. On the primary instance of Splunk Phantom, configure it to be the primary instance for warm standby. You are prompted to give passwords for the Splunk Phantom user, the PostgreSQL database replication user, and the information for creating a self-signed SSL certificate.
   
   phenv python /<PHANTOM_HOME>/bin/setup_warm_standby.pyc --primary-mode --configure --primary-ip <primary_ip> --standby-ip <standby_ip>
7. On the secondary instance, configure it to be the warm standby instance.
   
   phenv python /<PHANTOM_HOME>/bin/setup_warm_standby.pyc --standby-mode --configure --primary-ip <primary_ip> --standby-ip <standby_ip>
8. On the both instances of Splunk Phantom, verify that warm standby is replicating on each Splunk Phantom instance.
   
   phenv python /<PHANTOM_HOME>/bin/setup_warm_standby.pyc --status

Example output from Splunk Phantom primary:

========= Processed Params =========
  Instance looks like Primary
   DB replication configured with Standby set to:  <warm_standby_ip>/32
   DB replication currently streaming
   Vault sync configured
=========  Script Done =========

Example output from Phantom secondary or warm standby:

========= Processed Params =========
  Instance looks like Standby
   DB replication configured
   rsync configured
=========  Script Done =========

Restore a backup from a warm standby primary to a new Phantom instance

When you restore a backup of a Splunk Phantom warm standby primary to a new instance that you want to become the new primary, you must update the warm standby configuration and move several keys to the secondary instance.

To update the warm standby configuration, perform the following steps:

1. Open a terminal session for both the primary and secondary Splunk Phantom instances. Keep these sessions open until you complete these steps.
   1. From the command line, SSH to your primary Splunk Phantom instance.
      
      SSH <username>@<primary_phantom_hostname>
   2. SSH to your secondary and warm standby Splunk Phantom instance.
      
      SSH <username>@<warm_standby_phantom_hostname>
   3. In both sessions, elevate to root.
      
      sudo su -
2. On the primary instance of Splunk Phantom, perform the restore. See Restore Splunk Phantom from a backup.
3. On the primary instance of Splunk Phantom, disable warm standby.
   
   phenv python /<PHANTOM_HOME>/bin/setup_warm_standby.pyc --primary-mode --off
4. On the secondary instance of Splunk Phantom, disable warm standby.
   
   phenv python /<PHANTOM_HOME>/bin/setup_warm_standby.pyc --standby-mode --off
5. On the secondary instance of Splunk Phantom, stop all Splunk Phantom services.
   
   /<PHANTOM_HOME>/bin/stop_phantom.sh
   
   

   Failing to stop Splunk Phantom services on the secondary instance results in two active Splunk Phantom instances operating independently, polling for data and executing automated actions. This can result in data loss or other undesired results.

6. Copy these files from the new primary instance of Splunk Phantom to the secondary:
   1. /<PHANTOM_HOME>/keystore/private_key.pem
   2. /<PHANTOM_HOME>/www/phantom_ui/secret_key.py
   3. /<PHANTOM_HOME>/www/phantom_ui/secret_key.pyc
7. On the secondary instance of Splunk Phantom, set the permissions, ownership, and SELinux security contexts for the files you copied to the secondary.
   1. chmod 0640 /<PHANTOM_HOME>/keystore/private_key.pem /<PHANTOM_HOME>/phantom/www/phantom_ui/secret_key.py[c]
   2. chown root:phantom /<PHANTOM_HOME>/keystore/private_key.pem
   3. chown phantom:phantom /<PHANTOM_HOME>/www/phantom_ui/secret_key.py[c]
   4. restorecon /<PHANTOM_HOME>/keystore/private_key.pem /<PHANTOM_HOME>/www/phantom_ui/secret_key.py[c]
8. On the primary instance of Splunk Phantom, configure it to be the primary for warm standby. You are prompted to give passwords for the Splunk Phantom user, the PostgreSQL database replication user, and the information for creating a self-signed SSL certificate.
   
   phenv python /<PHANTOM_HOME>/bin/setup_warm_standby.pyc --primary-mode --configure --primary-ip <primary_ip> --standby-ip <standby_ip>
9. On the secondary instance, configure it to be the warm standby instance.
   
   phenv python /<PHANTOM_HOME>/bin/setup_warm_standby.pyc --standby-mode --configure --primary-ip <primary_ip> --standby-ip <standby_ip>
10. On both instances of Splunk Phantom, verify that the warm standby instance is replicating on each Phantom instance.
    
    phenv python /<PHANTOM_HOME>/bin/setup_warm_standby.pyc --status

Example output from Phantom primary:

========= Processed Params =========
  Instance looks like Primary
   DB replication configured with Standby set to:  <warm_standby_ip>/32
   DB replication currently streaming
   Vault sync configured
=========  Script Done =========

Example output from Phantom secondary or warm standby:

========= Processed Params =========
  Instance looks like Standby
   DB replication configured
   rsync configured
=========  Script Done =========