Warm standby feature overview
Warm standby is a strategy for high availability that regularly copies data from a primary instance of to a secondary instance. In the event of a failure on the primary, a systems administrator can quickly put the secondary into service as a new primary with minimal downtime or data loss.
's warm standby is implemented using PostgreSQL's streaming replication for the internal database and cron-based rsync of file system directories.
Warm standby is not a substitute for regular backups or other disaster recovery preparations.
Warm standby is configured in the same way, and works the same way on both privileged and unprivileged deployments.
Supported configurations
systems administrators can configure two identical, standalone instances to use the warm standby strategy.
- Only individual, standalone instances can be configured to use warm standby.
- instances with an external PostgreSQL database cannot use warm standby.
- clusters cannot use warm standby.
How to check the status of warm standby
You may need to check the status of warm standby before performing other actions, such as maintenance, upgrades, or troubleshooting. provides a command for checking the current status of warm standby.
To check the status of warm standby use the following command at the shell prompt.
Privileged instance
Unprivileged instance
The output will tell you which state your warm standby configuration is in. It will also tell you when the last WAL segment came in, and how many WAL segments are being held by the standby.
Warm standby is not configured on the instance
========= Processed Params ========= Warm Standby not yet configured ========= Script Done =========
Warm standby is configured as the primary and operating normally on the instance
========= Processed Params ========= Current Phantom Version: <phantom_version> Instance looks like Primary DB replication configured with Standby set to: <warm_standby_ip>/32 DB replication currently streaming WAL segments configured: 32 Vault sync configured ========= Script Done =========
Warm standby is configured as the secondary and operating normally on the instance
========= Processed Params ========= Current Phantom Version: <phantom_version> Instance looks like Standby DB replication configured DB last updated Wednesday <timestamp_last_updated> WAL segments in use: 2 WAL segments configured: 32 rsync configured ========= Script Done =========
IP addresses
Communication between the primary instance and the warm standby instance is handled using IP addresses, not hostnames. If the IP address changes for either instance, you must reestablish the warm standby pairing. See Create a warm standby.
PostgreSQL streaming replication
PostgreSQL streams data from its Write Ahead Log (WAL) from the primary database to the secondary database. This keeps the secondary synchronized with the primary database. See Streaming Replication on the PostgresSQL documentation site.
If network communication between the primary and secondary databases fails for any significant period of time, the WAL segments needed to update the secondary may have already been recycled by the primary. If this happens, you will need to recreate the warm standby relationship with the primary.
If your instances process high volumes of events or there is significant latency between the primary and the warm standby instance of , consider increasing the value of wal_keep_segments
in pg_hba.conf to a value large enough to prevent the primary from recycling WAL segments too quickly for your environment. This will use additional disk space on the primary instance, but can increase the reliability of the database synchronization.
Rsync
Rsync is used to synchronize file system data between the primary instance and the warm standby. When the setup_warm_standby.pyc script is run to set up warm standby, a cron job is created to run rsync to keep the warm standby up to date with the primary.
What is synchronized between the primary and the warm standby
File system directories synced from the primary to the warm standby.
Item to be synced | Directory or notes |
---|---|
PostgreSQL database | Synchronized using PostreSQL streaming replication. |
Files or Vault | /<PHANTOM_HOME>/vault |
Certificates and system files | /<PHANTOM_HOME>/etc |
Playbooks | /<PHANTOM_HOME>/scm/git/ |
App states | /<PHANTOM_HOME>/local_data/app_states |
Playbook data state | /<PHANTOM_HOME>/tmp |
SSL keys and certificates | /<PHANTOM_HOME>/etc/ssl
/<PHANTOM_HOME>/etc/ssl/certs |
Reports | /<PHANTOM_HOME>/vault/reports |
SAML configuration | /<PHANTOM_HOME>/www/phantom_ui/auth_backends/saml2_xml |
PIP and RPM packages are also synchronized. You can disable syncing PIP and RPM packages by using the --ignore-package-updates
option with setup_warm_standby.pyc.
Relevant log files Each of these log files can be useful when troubleshooting or maintaining your warm standby configuration.
Log file name | Path |
---|---|
Setup and command output | |
warm_standby.log | /var/log/phantom/warm_standby.log |
RSYNC state logs | |
rsync_opt.log | /var/log/phantom/rsync_opt.log |
rsync_vault.log | /var/log/phantom/rsync_vault.log |
rsync_apps.log | /var/log/phantom/rsync_apps.log |
rsync_app_states.log | /var/log/phantom/rsync_app_states.log |
rsync_etc_ssl_certs | /var/log/phantom/rsync_etc_ssl_certs |
rsync_playbooks.log | /var/log/phantom/rsync_playbooks.log |
rsync_reports.log | /var/log/phantom/rsync_reports.log |
rsync_pip_req.log | /var/log/phantom/rsync_pip_req.log |
rsync_rpm_req.log | /var/log/phantom/rsync_rpm_req.log |
Package installation logs | |
rpm_packages_primary.txt | /var/log/phantom/rpm_packages_primary.txt |
pip_packages_primary.txt | /var/log/phantom/pip_packages_primary.txt |
Use ibackup.pyc with warm standby | Create a warm standby |
This documentation applies to the following versions of Splunk® Phantom (Legacy): 4.9, 4.10, 4.10.1, 4.10.2, 4.10.3, 4.10.4, 4.10.6, 4.10.7
Feedback submitted, thanks!