Upgrade PostgreSQL for Splunk Phantom deployments with external databases

If your deployment of Splunk Phantom uses an externalized PostgreSQL database (or database cluster), you must upgrade the PostgresSQL database manually before you upgrade your Splunk Phantom instance or cluster.

PostgreSQL version 9.4 reached End of Life (EOL) in February 2020. Splunk Phantom now uses PostgreSQL 11.6.

Splunk Phantom 4.9 or later installs the standard version of PostgreSQL 11.6. Support for the Red Hat Source Collections version of PostgreSQL has been discontinued .

This topic is meant for the following cases:

• Your single instance Splunk Phantom deployment uses an externalized PostgreSQL database.
• Your Splunk Phantom cluster uses a Shared Services server, with its PostgreSQL database on that node.
• Your Splunk Phantom cluster uses a separate PostgreSQL server or a PostgreSQL cluster.

Prerequisites

Prior to upgrading PostgreSQL to version 11.6, do the following actions:

• Create a full backup of your PostgreSQL database. Store this backup in a safe place.
• If you want to preserve the logs from PostgreSQL 9.4, copy or back up the logs directory. For example to copy the database on a Shared Services server run:

  cp -avr /opt/phantom/data/db/pg_log/ /path/to/your/backup/location/

  • For a privileged instance or a Shared Services server the directory is /opt/phantom/data/db/pg_log.
  • For an unprivileged instance the directory is /<PHANTOM_HOME>/data/db/pg_log, where <PHANTOM_HOME> is the directory where Splunk Phantom is installed.
  • For external PostgreSQL databases or database clusters, the default directory is /var/lib/pgsql/9.4/data/pg_log/.
• Make a full backup of your Splunk Phantom deployment before upgrading. See Backup or restore your Splunk Phantom instance in Administer Splunk Phantom. Alternatively, for single instance deployments running as a virtual machine, create a snapshot of the virtual machine. Save this backup in a safe place.
• If your deployment is running a warm standby, disable warm standby. See Disable Warm Standby.
• If you are using automation to run ibackup.pyc tool, cancel backups that could run during your upgrade window. For example, if you have configured a cron job to run ibackup.pyc, disable that cron job.
• For a manual PostgreSQL upgrade, read the PostgreSQL upgrade instructions on postgresql.org.
• Finish the prerequisites to prepare your instance or cluster for upgrade from Splunk Phantom upgrade overview and prerequisites, including installing the Splunk Phantom repositories and signing keys packages.

Scripted PostgreSQL upgrade procedure

Splunk Phantom 4.9 includes a script for upgrading PostgreSQL which is installed when you install the repositories and signing keys, or unpack the installation TAR file. If during the upgrade procedure you get a "No such file or directory" error for one of the scripts, you need to install the Splunk Phantom repositories and signing keys packages.

Before you begin, decide if you want to upgrade leaving the directory for PostgreSQL the same as it was before upgrade (an in-place upgrade) or if you want the upgrade to put the upgraded database in a new directory.

After upgrading a default PostgreSQL installation, if you have not chosen an in-place upgrade, the database will be in a new directory such as /var/lib/pgsql/11.6/data/ rather than /var/lib/pgsql/9.4/data/. If you are upgrading a Shared Services server, the default database directory is /opt/phantom/data/db/.

1. Log in to the Splunk Phantom instance's operating system:
   • For privileged deployments, log in as the root user or a user with sudo privileges.
   • For unprivileged deployments, log in as the user account that runs Splunk Phantom.
2. On the Splunk Phantom instance, or on each cluster node, stop all Splunk Phantom services. For example, as the root user:

   /<PHANTOM_HOME>/bin/stop_phantom.sh

   For clustered deployments wait for each node to stop Splunk Phantom services before stopping services on the next node.

3. Use SCP to copy the file /<PHANTOM_HOME/bin/upgrade_postgresql_9.4_to_11.6.sh from a Splunk Phantom instance or cluster node to your PostgreSQL server. This file is installed when the Splunk Phantom repositories are installed via RPM or when the upgrade TAR file is extracted.
4. On the PostgreSQL server, as the root user, run the upgrade_postgresql_9.4_to_11.6.sh script.
   To put the new PostgreSQL database in a new directory:

   ./upgrade_postgresql_9.4_to_11.6.sh --old_data_path=<OLD_DATA_PATH> --new_data_path=<NEW_DATA_PATH>

   To put the new PostgreSQL database in the same directory as the previous version:

   ./upgrade_postgresql_9.4_to_11.6.sh --replace_data_path=<DATA_PATH>

   Replace the variables <OLD_DATA_PATH>, <NEW_DATA_PATH>, or <DATA_PATH> with the path or paths needed for your upgrade.
5. On the database server start PostgreSQL.

   systemctl start postgresql-11

6. On the Splunk Phantom instance, or on each cluster node, run the prepare_cluster_node_for_upgrade.sh script.
   
   • On each Splunk Phantom cluster node which installed the repositories using RPM:

     sudo phenv /opt/phantom/bin/prepare_cluster_node_for_upgrade.sh

   • On each Splunk Phantom cluster node which installed the repositories using the offline tar file:

     sudo phenv /usr/local/src/upgrade-<version>/bin/prepare_cluster_node_for_upgrade.sh

   • For an unprivileged instance or on each unprivileged cluster node:

     phenv ./bin/prepare_cluster_node_for_upgrade.sh

7. On the Splunk Phantom instance, or on each cluster node, start all Splunk Phantom services. If you are going to upgrade GlusterFS immediately after upgrading PostgreSQL, you can skip this step.
   As the root user:

   /opt/phantom/bin/start_phantom.sh

   Or for an unprivileged instance, as the user account that runs Splunk Phantom:

   /<PHANTOM_HOME>/bin/start_phantom.sh

   For clustered deployments wait for each node to completely start Splunk Phantom services before starting services on the next node.

Once the database upgrade has been successfully completed, you may proceed with upgrading either GlusterFS if you have not already completed that task, or with upgrading your Splunk Phantom instance or cluster nodes.

Manual PostgreSQL upgrade procedure

To manually upgrade PostgreSQL: See the upgrade instructions on postgresql.org.

After upgrading a default PostgreSQL installation, depending on the options chosen, the database may be in a new directory /var/lib/pgsql/11.6/data/ rather than /var/lib/pgsql/9.4/data/. If you are upgrading a Shared Services server, the database remains in /opt/phantom/data.

1. On your Splunk Phantom instance or each node of your Splunk Phantom cluster, stop Splunk Phantom.
   • Log in to the Splunk Phantom instance's operating system as either the root user or a user with sudo privileges.
   • If you're operating an unprivileged Splunk Phantom deployment, you will need to log in as the user that runs Splunk Phantom.
   • Stop all Splunk Phantom services. For example, as the root user:

     /opt/phantom/bin/stop_phantom.sh

     Or for an unprivileged instance, as the user account that runs Splunk Phantom:

     /<PHANTOM_HOME>/bin/stop_phantom.sh

     For clustered deployments wait for each node to stop Splunk Phantom services before stopping services on the next node.

2. Log in to the PostgreSQL database server and perform the upgrade by following the procedure at PostgreSQL upgrade instructions on postgresql.org.
3. On the database server start PostgreSQL.

   systemctl start postgresql-11

4. On the Splunk Phantom instance, or on each cluster node, run the prepare_cluster_node_for_upgrade.sh script.
   
   • On each Splunk Phantom cluster node which installed the repositories using RPM:

     sudo phenv /opt/phantom/bin/prepare_cluster_node_for_upgrade.sh

   • On each Splunk Phantom cluster node which installed the repositories using the offline tar file:

     sudo phenv /usr/local/src/upgrade-<version>/bin/prepare_cluster_node_for_upgrade.sh

   • For an unprivileged instance or on each unprivileged cluster node:

     phenv ./bin/prepare_cluster_node_for_upgrade.sh

     If you get a "No such file or directory" error for this script, you need to install the Splunk Phantom repositories and signing keys packages.

5. On the Splunk Phantom instance, or on each cluster node, start all Splunk Phantom services. If you are going to upgrade GlusterFS immediately after upgrading PostgreSQL, you can skip this step.
   For example, as the root user:

   /opt/phantom/bin/start_phantom.sh

   Or for an unprivileged instance, as the user account that runs Splunk Phantom:

   /<PHANTOM_HOME>/bin/start_phantom.sh

   For clustered deployments wait for each node to completely start Splunk Phantom services before starting services on the next node.

Once the database upgrade has been successfully completed, you may proceed with upgrading either GlusterFS if you have not already completed that task, or with upgrading your Splunk Phantom instance or cluster nodes.