Warm standby feature overview
Warm standby is a strategy for high availability that regularly copies data from a primary instance of Splunk Phantom to a secondary instance. In the event of a failure on the primary, a systems administrator can quickly put the secondary into service as a new primary with minimal downtime or data loss.
Splunk Phantom's warm standby is implemented using PostgreSQL's streaming replication for the internal database and cron-based rsync of file system directories.
Warm standby is not a substitute for regular backups or other disaster recovery preparations.
Splunk Phantom systems administrators can configure two identical, standalone Splunk Phantom instances to use the warm standby strategy.
- Only individual, standalone Splunk Phantom instances can be configured to use warm standby.
- Splunk Phantom instances with an external PostgreSQL database cannot use warm standby.
- Splunk Phantom clusters cannot use warm standby.
How to check the status of warm standby
You may need to check the status of warm standby before performing other actions, such as maintenance, upgrades, or troubleshooting. Splunk Phantom provides a command for checking the current status of warm standby.
To check the status of warm standby use the following command at the shell prompt.
The output will tell you which state your warm standby configuration is in.
Warm standby is not configured on the instance
========= Processed Params ========= Warm Standby not yet configured ========= Script Done =========
Warm standby is configured as the primary and operating normally on the instance
========= Processed Params ========= Instance looks like Primary DB replication configured with Standby set to: <warm_standby_ip>/32 DB replication currently streaming Vault sync configured ========= Script Done =========
Warm standby is configured as the secondary and operating normally on the instance
========= Processed Params ========= Instance looks like Standby DB replication configured rsync configured ========= Script Done =========
Communication between the primary Splunk Phantom instance and the warm standby instance is handled using IP addresses, not hostnames. If the IP address changes for either instance, you must reestablish the warm standby pairing. See Create a warm standby.
PostgreSQL streaming replication
PostgreSQL streams data from its Write Ahead Log (WAL) from the primary database to the secondary database. This keeps the secondary synchronized with the primary database. See Streaming Replication on the PostgresSQL documentation site.
If network communication between the primary and secondary databases fails for any significant period of time, the WAL segments needed to update the secondary may have already been recycled by the primary. If this happens, you will need to recreate the warm standby relationship with the primary.
If your Splunk Phantom instances process high volumes of events or there is significant latency between the primary and the warm standby instance of Splunk Phantom, consider increasing the value of
wal_keep_segments in pg_hba.conf to a value large enough to prevent the primary from recycling WAL segments too quickly for your environment. This will use additional disk space on the primary Splunk Phantom instance, but can increase the reliability of the database synchronization.
Rsync is used to synchronize file system data between the primary Splunk Phantom instance and the warm standby. When the setup_warm_standby.pyc script is run to set up warm standby, a cron job is created to run rsync to keep the warm standby up to date with the primary.
What is synchronized between the primary and the warm standby
File system directories synced from the primary to the warm standby.
|Item to be synced||Directory or notes|
|PostgreSQL database||Synchronized using PostreSQL streaming replication.|
|Files or Vault||/<PHANTOM_HOME>/vault|
|Certificates and system files||/<PHANTOM_HOME>/etc|
|Playbook data state||/<PHANTOM_HOME>/tmp|
|SSL keys and certificates||/<PHANTOM_HOME>/etc/ssl
PIP and RPM packages are also synchronized. You can disable syncing PIP and RPM packages by using the
--ignore-package-updates option with setup_warm_standby.pyc.
Relevant log files Each of these log files can be useful when troubleshooting or maintaining your warm standby configuration.
|Log file name||Path|
|Setup and command output|
|RSYNC state logs|
|Package installation logs|
Use ibackup.pyc with warm standby
Create a warm standby
This documentation applies to the following versions of Splunk® Phantom: 4.8