Back up a deployment

Your deployment is backed up using command-line tools. You do not need to take offline to do a backup.

Before you begin, check each of the following things.

• For privileged deployments, make sure that you have root or sudo permissions on the deployment.
• Make sure you have completed either the section Prepare for a back up or Prepare a cluster or deployment with an external PostgreSQL database, depending on your deployment's configuration.

Deployments of Splunk Phantom in AWS that use RDS for their PostgreSQL database should not use ibackup.pyc to create backups of the Phantom database. Use the backup.pyc tool instead. See Create a full backup for deployments with an external PostgreSQL database in RDS.

Prepare for a backup

Before you can create backups, you must prepare by running the ibackup.pyc command with the --setup option.

You must prepare by running the ibackup.pyc command with the --setup option after any upgrade of .

During setup, ibackup.pyc temporarily stops services in order to make changes to the PostgreSQL database configuration. Services restart before setup is complete. This causes the following error to be displayed during the setup process, which can be ignored:

psql: ERROR:  pgbouncer cannot connect to server

If your Splunk Phantom deployment uses a Warm Standby or if you intend to use a Warm Standby, you must configure Warm Standby before setting up backups.

1. From the command line, SSH to your instance.
   
   ssh <username>@<phantom_hostname>
2. Prepare the system for a backup.
   
   /opt/phantom/bin/phenv ibackup --setup

The command output looks like this:

[user@phantom bin]$ phenv python ibackup.pyc --setup
[11/Dec/2019 19:11:14] INFO: Running ibackup.pyc - details will be logged to /var/log/phantom/backup/backup_2019-12-11T19:11:14.562281Z.log
[11/Dec/2019 19:11:14] INFO: Attempting to connect to Postgresql ...
Setup will temporarily stop phantom. If you wish to continue, enter yes to proceed:yes
[11/Dec/2019 19:11:38] INFO: Attempting to connect to Postgresql ...
psql: ERROR:  pgbouncer cannot connect to server
[11/Dec/2019 19:11:38] INFO: Retrying ...
[11/Dec/2019 19:11:39] INFO: Attempting to connect to Postgresql ...
psql: ERROR:  pgbouncer cannot connect to server
[11/Dec/2019 19:11:39] INFO: Retrying ...
[11/Dec/2019 19:11:41] INFO: Attempting to connect to Postgresql ...
psql: ERROR:  pgbouncer cannot connect to server
[11/Dec/2019 19:11:41] INFO: Retrying ...
[11/Dec/2019 19:11:45] INFO: Attempting to connect to Postgresql ...
psql: ERROR:  pgbouncer cannot connect to server
[11/Dec/2019 19:11:45] INFO: Retrying ...
[11/Dec/2019 19:11:53] INFO: Attempting to connect to Postgresql ...
[11/Dec/2019 19:11:53] INFO: Setup complete

If the instance or cluster has been setup for backups, the tool prompts you to confirm that you wish to run setup again:

You appear to have already run setup. Re-run again ? Enter yes to proceed:

Enter yes to run the setup process again.

Running the set up again after a restore resets the state file used by ibackup.pyc and archives all existing backup files. Running it a second time before any restore actions does nothing.

Prepare a cluster or deployment with an external PostgreSQL database for a backup

For deployments where the PostgreSQL database is external to the instance or in clustered deployments, you must take additional steps to prepare the deployment for backup.

In clustered deployments, all backup commands must be issued from the same cluster node.

1. From the command line, SSH to one cluster node of your deployment.
   
   ssh <username>@<phantom_hostname>
2. Run ibackup.pyc. This step generates the db_bootstrap.tgz file.
   
   /opt/phantom/bin/phenv ibackup --setup
3. Use SCP to copy db_bootstrap.tgz to your PostgreSQL database host.
   
   scp db_bootstrap.tgz <username>@<postgresql_hostname>:/<directory>
4. SSH to the PostgreSQL database host.
   
   ssh <username>@<postgresql_hostname>
5. Extract the db_bootstrap.tgz file.
   
   tar -xzvf db_bootstrap.tgz
6. Change directory to the setup directory created when db_bootstrap.tgz was extracted.
   
   cd setup
7. Run the extdb_backup_bootstrap script with the parameter --pgdata </path/to/postgresql/db>.
   For version 4.8:
   python extdb_backup_bootstrap.pyc --pgdata </path/to/postgresql/db>
   For version 4.9 and later:
   python extdb_backup_bootstrap.py --pgdata </path/to/postgresql/db>
8. SSH to the cluster node where you ran ibackup.pyc.
   
   ssh <username>@<phantom_hostname>
9. Run ibackup.pyc to complete the setup.
   
   /opt/phantom/bin/phenv ibackup --setup

Create a full backup

Your instance is backed up using command-line tools. You do not need to take offline to create a backup.

In clustered deployments, all backup commands must be issued from the same cluster node.

1. From the command line, SSH to your instance.
   ssh <username>@<phantom_hostname>
2. Run the following command to perform the backup.
   phenv python ibackup.pyc --backup --backup-type full
   If no backups exist, a full backup is created. When a full backup exists, a new incremental backup is added to the group. If you already have a backup group and want to create a new full backup, use the --backup-type full argument.

The following sample output shows a backup being run:

[11/Dec/2019 21:24:28] INFO: Running ibackup.pyc - details will be logged to /var/log/phantom/backup/backup_2019-12-11T21:24:28.706665Z.log
[11/Dec/2019 21:24:28] INFO: Attempting to connect to Postgresql ...
[11/Dec/2019 21:24:30] INFO: First backup. Performing full backup
[11/Dec/2019 21:24:30] INFO: Backing up files
[11/Dec/2019 21:25:03] INFO: Backing up database
[11/Dec/2019 21:25:17] WARNING: no prior backup exists, incr backup has been changed to full

[11/Dec/2019 21:25:58] INFO: Backup created at: /opt/phantom/data/backup/phantom_backup_group_0_level_0.tar
[11/Dec/2019 21:25:58] INFO: You should ensure this tarball is kept safe. It will be required for restore

If you receive an error that postgresql.conf is owned by the root user, you will need to use the chown and chgrp commands to set the ownership of /<PHANTOM_HOME>/phantom/data/db/postgresql.conf to the postgres user. For clustered deployments, do this on the database node of the deployment.

Create a full backup for deployments with an external PostgreSQL database in RDS

Amazon Web Services RDS provides automatic backups of hosted PostgreSQL databases which are managed and restored using the management console. See Backing up and restoring an Amazon RDS DB instance in the AWS documentation.

To back up other components, use the older backup.pyc tool.

1. From the command line, SSH to your instance.
   
   ssh <username>@<phantom_hostname>
2. Change the directory to <PHANTOM HOME>/bin.
   
   cd <PHANTOM_HOME>/bin
3. Perform the backup.
   
   phenv python backup.pyc --all
   In this case, you can safely ignore the deprecation warning.

The command output looks like this:

[root@phantom bin]# phenv python backup.pyc --all
[pid:  8548] [09/Sep/2020 22:49:17]    backup.py:609  WARNING: The --all option of the backup.pyc script has been deprecated. Please use the ibackup.pyc script to perform backups and restores. Documentation for the new script can be found at https://docs.splunk.com/Documentation/Phantom/4.9/Admin/BackupOrRestoreOverview. The --config option of the backup.pyc script will continue to work with ibackup.pyc.


[2020-09-09 22:49:17] The --all option of the backup.pyc script has been deprecated. Please use the ibackup.pyc script to perform backups and restores. Documentation for the new script can be found at https://docs.splunk.com/Documentation/Phantom/4.9/Admin/BackupOrRestoreOverview. The --config option of the backup.pyc script will continue to work with ibackup.pyc.


backup.pyc 2.0.0
[2020-09-09 22:49:17] Stopping all Phantom services except PostgreSQL
Stopping Supervisor daemon manager...[  OK  ]
/opt/phantom/proxy/bin/splunk_proxyd is already stopped
Stopping phantom_actiond:                                  [  OK  ]
Stopping phantom_workflowd:                                [  OK  ]
Stopping phantom_ingestd:                                  [  OK  ]
Stopping phantom_decided:                                  [  OK  ]
Stopping splunkd...
Shutting down.  Please wait, as this may take a few minutes.
 [  OK  ]
Stopping splunk helpers...
 [  OK  ]
Done.
[2020-09-09 22:49:28] Backing up Phantom DB
[===========================================================================] 100%
[2020-09-09 22:49:29] Generating CSVs for configuration tables
[===========================================================================] 100%
[2020-09-09 22:49:31] Validating CSVs
[2020-09-09 22:49:31] Backing up required Phantom directories
[===========================================================================] 100%
[2020-09-09 22:50:37] Backing up Phantom-specific files
[===========================================================================] 100%
[2020-09-09 22:50:37] Compressing backup (this may take a while)
[2020-09-09 22:50:47] Running /opt/phantom/bin/start_phantom.sh
Starting all Phantom services
Phantom startup successful
logs recorded in /opt/phantom/data/phantom_backups/phantom_backup_2020-09-09-22-49-17.log
Backup located at /opt/phantom/data/phantom_backups/phantom_backup_2020-09-09-22-49-17.tgz

Create an incremental backup

The ibackup.pyc tool checks whether a full backup exists. If a full backup doesn't exist, a full backup is created. If a full backup exists, an incremental backup is created and added to the backup chain. You do not need to take offline to create a backup. If your deployment's PostgreSQL database is hosted in Amazon's RDS, you cannot use incremental backups for the database.

In clustered deployments, all backup commands must be issued from the same cluster node.

1. From the command line, SSH to your instance.
   
   ssh <username>@<phantom_hostname>
2. Change the directory to <PHANTOM HOME>/bin.
   
   cd <PHANTOM_HOME>/bin
3. Perform the backup.
   
   /opt/phantom/bin/phenv ibackup --backup

The command output looks like this:

[11/Dec/2019 21:29:47] INFO: Running ibackup.pyc - details will be logged to /var/log/phantom/backup/backup_2019-12-11T21:29:47.549233Z.log
[11/Dec/2019 21:29:47] INFO: Attempting to connect to Postgresql ...
[11/Dec/2019 21:29:53] INFO: Backing up files
[11/Dec/2019 21:29:54] INFO: Backing up database
[11/Dec/2019 21:30:03] INFO: Backup created at: /opt/phantom/data/backup/phantom_backup_group_0_level_1.tar
[11/Dec/2019 21:30:03] INFO: You should ensure this tarball is kept safe. It will be required for restore

You can override the default behavior by using this command-line option:

/opt/phantom/bin/phenv ibackup --backup --backup-type incr

Save your backups in a safe place, such as one that is not on the same disk, partition, or virtual machine as your Splunk Phantom instance.