Warm standby feature overview

Warm standby is a strategy for high availability that regularly copies data from a primary instance of Splunk Phantom to a secondary instance. In the event of a failure on the primary, a systems administrator can quickly put the secondary into service as a new primary with minimal downtime or data loss.

Splunk Phantom's warm standby is implemented using PostgreSQL's streaming replication for the internal database and cron-based rsync of file system directories.

Warm standby is not a substitute for regular backups or other disaster recovery preparations.

Supported configurations

Splunk Phantom systems administrators can configure two identical, standalone Splunk Phantom instances to use the warm standby strategy.

• Only individual, standalone Splunk Phantom instances can be configured to use warm standby.
• Splunk Phantom instances with an external PostgreSQL database cannot use warm standby.
• Splunk Phantom clusters cannot use warm standby.

How to check the status of warm standby

You may need to check the status of warm standby before performing other actions, such as maintenance, upgrades, or troubleshooting. Splunk Phantom provides a command for checking the current status of warm standby.

To check the status of warm standby use the following command at the shell prompt.

Privileged instance

phenv python2.7 /<PHANTOM_HOME>/bin/setup_warm_standby.pyc --status

Unprivileged instance

phenv python2.7 /<PHANTOM_HOME>/bin/setup_warm_standby.pyc --status

The output will tell you which state your warm standby configuration is in.

Warm standby is not configured on the instance

 ========= Processed Params =========
   Warm Standby not yet configured
 =========  Script Done =========

Warm standby is configured as the primary and operating normally on the instance

 ========= Processed Params =========
   Instance looks like Primary
    DB replication configured with Standby set to:  <warm_standby_ip>/32
    DB replication currently streaming
    Vault sync configured
 =========  Script Done =========

Warm standby is configured as the secondary and operating normally on the instance

 ========= Processed Params =========
   Instance looks like Standby
    DB replication configured
    rsync configured
 =========  Script Done =========

IP addresses

Communication between the primary Splunk Phantom instance and the warm standby instance is handled using IP addresses, not hostnames. If the IP address changes for either instance, you must reestablish the warm standby pairing. See Create a warm standby.

PostgreSQL streaming replication

PostgreSQL streams data from its Write Ahead Log (WAL) from the primary database to the secondary database. This keeps the secondary synchronized with the primary database. See Streaming Replication on the PostgresSQL documentation site.

If network communication between the primary and secondary databases fails for any significant period of time, the WAL segments needed to update the secondary may have already been recycled by the primary. If this happens, you will need to recreate the warm standby relationship with the primary.

If your Splunk Phantom instances process high volumes of events or there is significant latency between the primary and the warm standby instance of Splunk Phantom, consider increasing the value of wal_keep_segments in pg_hba.conf to a value large enough to prevent the primary from recycling WAL segments too quickly for your environment. This will use additional disk space on the primary Splunk Phantom instance, but can increase the reliability of the database synchronization.

Rsync

Rsync is used to synchronize file system data between the primary Splunk Phantom instance and the warm standby. When the setup_warm_standby.pyc script is run to set up warm standby, a cron job is created to run rsync to keep the warm standby up to date with the primary.

What is synchronized between the primary and the warm standby

File system directories synced from the primary to the warm standby.

Item to be synced	Directory or notes
PostgreSQL database	Synchronized using PostreSQL streaming replication.
Files or Vault	/<PHANTOM_HOME>/vault
Certificates and system files	/<PHANTOM_HOME>/etc
Playbooks	/<PHANTOM_HOME>/scm/git/
App states	/<PHANTOM_HOME>/local_data/app_states
Playbook data state	/<PHANTOM_HOME>/tmp
SSL keys and certificates	/<PHANTOM_HOME>/etc/ssl /<PHANTOM_HOME>/etc/ssl/certs
Reports	/<PHANTOM_HOME>/vault/reports
SAML configuration	/<PHANTOM_HOME>/www/phantom_ui/auth_backends/saml2_xml

PIP and RPM packages are also synchronized. You can disable syncing PIP and RPM packages by using the --ignore-package-updates option with setup_warm_standby.pyc.

Relevant log files Each of these log files can be useful when troubleshooting or maintaining your warm standby configuration.

Log file name	Path
Setup and command output
warm_standby.log	/var/log/phantom/warm_standby.log
RSYNC state logs
rsync_opt.log	/var/log/phantom/rsync_opt.log
rsync_vault.log	/var/log/phantom/rsync_vault.log
rsync_apps.log	/var/log/phantom/rsync_apps.log
rsync_app_states.log	/var/log/phantom/rsync_app_states.log
rsync_etc_ssl_certs	/var/log/phantom/rsync_etc_ssl_certs
rsync_playbooks.log	/var/log/phantom/rsync_playbooks.log
rsync_reports.log	/var/log/phantom/rsync_reports.log
rsync_pip_req.log	/var/log/phantom/rsync_pip_req.log
rsync_rpm_req.log	/var/log/phantom/rsync_rpm_req.log
Package installation logs
rpm_packages_primary.txt	/var/log/phantom/rpm_packages_primary.txt
pip_packages_primary.txt	/var/log/phantom/pip_packages_primary.txt