Warm standby feature overview
Warm standby is a strategy for high availability that regularly copies data from a primary instance of to a secondary instance. In the event of a failure on the primary, a systems administrator can quickly put the secondary into service as a new primary with minimal downtime or data loss.
's warm standby is implemented using PostgreSQL's streaming replication for the internal database and cron-based rsync of file system directories.
Warm standby is not a substitute for regular backups or other disaster recovery preparations.
systems administrators can configure two identical, standalone instances to use the warm standby strategy.
- Only individual, standalone instances can be configured to use warm standby.
- instances with an external PostgreSQL database cannot use warm standby.
- clusters cannot use warm standby.
If your deployment uses an external PostgreSQL 12.9 database, you cannot use the warm standby feature.
As a best practice, ensure that the /opt/phantom directory has two to three times the amount of free disk space that is currently used by the database before running the setup as this command creates a backup of the data in <PHANTOM_HOME>/data/db. This command then copies the data to what you have set your --recovery-database-location to, which by default is <PHANTOM_HOME>/tmp/db. If the required disk space is not available for the backup, the command fails.
If you use the --ignore-database argument, you won't need as much free disk space as only system files are backed up. See Warm standby tools.
How to check the status of warm standby
You may need to check the status of warm standby before performing other actions, such as maintenance, upgrades, or troubleshooting. provides a command for checking the current status of warm standby.
To check the status of warm standby use the following command at the shell prompt.
The output will tell you which state your warm standby configuration is in. It will also tell you when the last WAL segment came in, and how many WAL segments are being held by the standby.
Warm standby is not configured on the instance
========= Processed Params ========= Warm Standby not yet configured ========= Script Done =========
Warm standby is configured as the primary and operating normally on the instance
========= Processed Params ========= Current Phantom Version: <phantom_version> Instance looks like Primary DB replication configured with Standby set to: <warm_standby_ip>/32 DB replication currently streaming WAL segments configured: 32 Vault sync configured ========= Script Done =========
Warm standby is configured as the secondary and operating normally on the instance
========= Processed Params ========= Current Phantom Version: <phantom_version> Instance looks like Standby DB replication configured DB last updated Wednesday <timestamp_last_updated> WAL segments in use: 2 WAL segments configured: 32 rsync configured ========= Script Done =========
Communication between the primary instance and the warm standby instance is handled using IP addresses, not hostnames. If the IP address changes for either instance, you must reestablish the warm standby pairing. See Create a warm standby.
PostgreSQL streaming replication
PostgreSQL streams data from its Write Ahead Log (WAL) from the primary database to the secondary database. This keeps the secondary synchronized with the primary database. See Streaming Replication on the PostgresSQL documentation site.
If network communication between the primary and secondary databases fails for any significant period of time, the WAL segments needed to update the secondary may have already been recycled by the primary. If this happens, you will need to recreate the warm standby relationship with the primary.
If your instances process high volumes of events or there is significant latency between the primary and the warm standby instance of , consider increasing the value of
wal_keep_segments to a value large enough to prevent the primary from recycling WAL segments too quickly for your environment. This will use additional disk space on the primary instance, but can increase the reliability of the database synchronization. See Manage warm standby features and options.
Rsync is used to synchronize file system data between the primary instance and the warm standby. When the setup_warm_standby.pyc script is run to set up warm standby, a cron job is created to run rsync to keep the warm standby up to date with the primary.
What is synchronized between the primary and the warm standby
File system directories synced from the primary to the warm standby.
|Item to be synced||Directory or notes|
|PostgreSQL database||Synchronized using PostreSQL streaming replication.|
|Files or Vault||/<PHANTOM_HOME>/vault|
|Certificates and system files||/<PHANTOM_HOME>/etc|
|Playbook data state||/<PHANTOM_HOME>/tmp|
|SSL keys and certificates||/<PHANTOM_HOME>/etc/ssl
PIP and RPM packages are also synchronized. You can disable syncing PIP and RPM packages by using the
--ignore-package-updates option with setup_warm_standby.pyc.
Relevant log files
Each of these log files can be useful when troubleshooting or maintaining your warm standby configuration.
|Log file name||Path|
|Setup and command output|
|RSYNC state logs|
|Package installation logs|
Use ibackup with warm standby
Create a warm standby
This documentation applies to the following versions of Splunk® SOAR (On-premises): 5.4.0, 5.5.0, 6.0.0, 6.0.1, 6.0.2, 6.1.0, 6.1.1, 6.2.0