Warm standby feature overview

Warm standby is a strategy for high availability that regularly copies data from a primary instance of to a secondary instance. In the event of a failure on the primary, a systems administrator can quickly put the secondary into service as a new primary with minimal downtime or data loss.

's warm standby is implemented using PostgreSQL's streaming replication for the internal database and cron-based rsync of file system directories.

Warm standby is not a substitute for regular backups or other disaster recovery preparations.

Supported configurations

systems administrators can configure two identical, standalone instances to use the warm standby strategy.

• Only individual, standalone instances can be configured to use warm standby.
• instances with an external PostgreSQL database cannot use warm standby.
• clusters cannot use warm standby.

If your deployment uses an external PostgreSQL 12.9 database, you cannot use the warm standby feature.

Storage

As a best practice, ensure that the /opt/phantom directory has two to three times the amount of free disk space that is currently used by the database before running the setup as this command creates a backup of the data in <PHANTOM_HOME>/data/db. This command then copies the data to what you have set your --recovery-database-location to, which by default is <PHANTOM_HOME>/tmp/db. If the required disk space is not available for the backup, the command fails.

If you use the --ignore-database argument, you won't need as much free disk space as only system files are synced. See Warm standby tools.

How to check the status of warm standby

You may need to check the status of warm standby before performing other actions, such as maintenance, upgrades, or troubleshooting. provides a command for checking the current status of warm standby.

To check the status of warm standby use the following command at the shell prompt.

phenv python /<PHANTOM_HOME>/bin/setup_warm_standby.pyc --status

The output will tell you which state your warm standby configuration is in. It will also tell you when the last WAL segment came in, and how many WAL segments are being held by the standby.

Warm standby is not configured on the instance

 ========= Processed Params =========
   Warm Standby not yet configured
 =========  Script Done =========

Warm standby is configured as the primary and operating normally on the instance

  ========= Processed Params =========
  Current Phantom Version: <phantom_version>
   Instance looks like Primary
    DB replication configured with Standby set to:  <warm_standby_ip>/32
    DB replication currently streaming
    WAL segments configured: 32
    Vault sync configured
  =========  Script Done =========

Warm standby is configured as the secondary and operating normally on the instance

  ========= Processed Params =========
  Current Phantom Version: <phantom_version>
   Instance looks like Standby
    DB replication configured
    DB last updated Wednesday <timestamp_last_updated>
    WAL segments in use: 2
    WAL segments configured: 32
    rsync configured
  =========  Script Done =========

IP addresses

Communication between the primary instance and the warm standby instance is handled using IP addresses, not hostnames. If the IP address changes for either instance, you must reestablish the warm standby pairing. See Create a warm standby.

PostgreSQL streaming replication

PostgreSQL streams data from its Write Ahead Log (WAL) from the primary database to the secondary database. This keeps the secondary synchronized with the primary database. See Streaming Replication on the PostgresSQL documentation site.

If network communication between the primary and secondary databases fails for any significant period of time, the WAL segments needed to update the secondary may have already been recycled by the primary. If this happens, you will need to recreate the warm standby relationship with the primary.

If your instances process high volumes of events or there is significant latency between the primary and the warm standby instance of , consider increasing the value of wal_keep_segments to a value large enough to prevent the primary from recycling WAL segments too quickly for your environment. This will use additional disk space on the primary instance, but can increase the reliability of the database synchronization. See Manage warm standby features and options.

Rsync

Rsync is used to synchronize file system data between the primary instance and the warm standby. When the setup_warm_standby.pyc script is run to set up warm standby, a cron job is created to run rsync to keep the warm standby up to date with the primary.

What is synchronized between the primary and the warm standby

File system directories synced from the primary to the warm standby.

Item to be synced	Directory or notes
PostgreSQL database	Synchronized using PostreSQL streaming replication.
Files or Vault	/<PHANTOM_HOME>/vault
Certificates and system files	/<PHANTOM_HOME>/etc
Playbooks	/<PHANTOM_HOME>/scm/git/
App states	/<PHANTOM_HOME>/local_data/app_states
Playbook data state	/<PHANTOM_HOME>/tmp
SSL keys and certificates	/<PHANTOM_HOME>/etc/ssl /<PHANTOM_HOME>/etc/ssl/certs
Reports	/<PHANTOM_HOME>/vault/reports
SAML configuration	/<PHANTOM_HOME>/www/phantom_ui/auth_backends/saml2_xml

PIP and RPM packages are also synchronized. You can use the --ignore-package-updates option with setup_warm_standby.pyc if you don't want to reinstall PIP and RPM packages on the standby instance and as such skip the PIP and RPM package synchronization of the primary and standby instances.

Relevant log files

Each of these log files can be useful when troubleshooting or maintaining your warm standby configuration.

Log file name	Path
Setup and command output
warm_standby.log	<PHANTOM_HOME>/var/log/phantom/warm_standby.log
RSYNC state logs
rsync_opt.log	<PHANTOM_HOME>/var/log/phantom/rsync_opt.log
rsync_vault.log	<PHANTOM_HOME>/var/log/phantom/rsync_vault.log
rsync_apps.log	<PHANTOM_HOME>/var/log/phantom/rsync_apps.log
rsync_app_states.log	<PHANTOM_HOME>/var/log/phantom/rsync_app_states.log
rsync_etc_ssl_certs	<PHANTOM_HOME>/var/log/phantom/rsync_etc_ssl_certs
rsync_playbooks.log	<PHANTOM_HOME>/var/log/phantom/rsync_playbooks.log
rsync_reports.log	<PHANTOM_HOME>/var/log/phantom/rsync_reports.log
rsync_pip_req.log	<PHANTOM_HOME>/var/log/phantom/rsync_pip_req.log
rsync_rpm_req.log	<PHANTOM_HOME>/var/log/phantom/rsync_rpm_req.log
Package installation logs
rpm_packages_primary.txt	<PHANTOM_HOME>/var/log/phantom/rpm_packages_primary.txt
pip_packages_primary.txt	<PHANTOM_HOME>/var/log/phantom/pip_packages_primary.txt