Upgrade a distributed RHEL installation of Splunk UBA

Perform the following steps to upgrade a distributed RHEL installation of Splunk UBA.

Before you begin

Make sure the correct hadoop ports are open. See, Inbound networking port requirements.

Before upgrading Splunk UBA in a distributed deployment, confirm you meet the Upgrade Splunk UBA prerequisites. Make sure that the prerequisites are verified on each server in the distributed deployment.

If you have enabled the integration that sends UBA audit data to Splunk Enterprise Security (ES), preserve the associated certificate before upgrading. For more info on sending audit data to ES see, Send Splunk UBA audit events to Splunk ES. Use the following command to export the certificate:

. /opt/caspida/bin/CaspidaCommonEnv.sh
sudo keytool -exportcert -alias "splunk es" -keystore $JAVA_HOME/lib/security/cacerts -rfc -file ~/splunk-es_cacert.pem

If you have enabled validation of the SSL certificate from Splunk ES, export the SSL certificate used for validating datasources from the Splunk Enterprise platform:

. /opt/caspida/bin/CaspidaCommonEnv.sh
sudo keytool -exportcert -alias "SplunkESRootCA" -keystore $JAVA_HOME/lib/security/cacerts -rfc -file ~/SplunkESRootCA.pem

Download the Splunk UBA upgrade software

Perform the following steps to find and download the Splunk UBA upgrade software:

1. Obtain the Splunk UBA Software Update and download the file to the /home/caspida directory on the management node. Select version 5.2.0 from the drop-down list. The downloadable archive file is named splunk-uba-software-upgrade-package_520.tgz.
2. Extract the archive with the following command:

   tar xfz /home/caspida/splunk-uba-software-upgrade-package_520.tgz -C /home/caspida

   The following files are extracted:

   • Splunk-UBA-Platform-5.2.0-20230202-8254784.tgz
   • Splunk-UBA-Platform-5.2.0-20230202-8254784.tgz.md5sum
   • uba-ext-pkgs-5.2.0.tgz
   • uba-ext-pkgs-5.2.0.tgz.md5sum

Stop all UBA services and archive

Switch to caspida user and perform the following steps to archive the older version of UBA.

1. On the management node, stop all the caspida services:

   /opt/caspida/bin/Caspida stop-all

2. On each node, archive the older version of UBA. REPLACE <old-uba-version> with your UBA version:

   sudo mkdir -p /var/vcap/release_archives
   sudo mv /opt/caspida /var/vcap/release_archives/caspida-<old-uba-version>
   sudo mkdir -p /opt/caspida && sudo chown caspida:caspida /opt/caspida
   sudo chown caspida:caspida /var/vcap/release_archives
   sudo chown caspida:caspida /var/vcap/release_archives/caspida-<old-uba-version>

Upgrade from RHEL 7.8 to RHEL 7.9

On each node, perform the following steps to upgrade your RHEL operating system from version 7.8 to version 7.9. If you are already using RHEL 7.9, go to Upgrade from RHEL 7.9 to 8.6.

1. Log in to the Splunk UBA management node as the caspida user.
2. In the /usr/lib/jvm/java-1.8.0/jre/lib/security/java.security file, comment out the following code:

   # jdk.tls.disabledAlgorithms=MD5, SSLv3, DSA, RSA keySize < 2048
   # jdk.tls.disabledAlgorithms=SSLv3, TLSv1, TLSv1.1, RC4, DES, MD5withRSA, \
   # DH keySize < 1024, EC keySize < 224, 3DES_EDE_CBC, anon, NULL, \
   # include jdk.disabled.namedCurves
   

3. Run the following command:

   sudo yum update -y

Upgrade from RHEL 7.9 to RHEL 8.6

Perform the following steps to upgrade your RHEL operating system from version 7.9 to version 8.6.

1. On each node, perform the OS upgrade steps as the root user.
2. Perform the initial checks of the system:
1. Check that the machine is properly subscribed to the RHEL repositories:

   subscription-manager list --installed

2. Run the following commands to make sure the required repositories are enabled during the upgrade:

   subscription-manager repos --enable rhel-7-server-rpms
   subscription-manager repos --enable rhel-7-server-extras-rpms
   

3. Run the following command to set the Red Hat Subscription Manager to consume the latest RHEL 7 content:

   subscription-manager release --unset

4. Set system locale to en_US.UTF-8 in the /etc/locale.conf file:

   LANG="en_US.UTF-8"
   LC_ALL="en_US.UTF-8"
   LC_CTYPE="en_US.UTF-8"
   

5. Run the following command to source the /etc/locale.conf file:

   source /etc/locale.conf

6. Run the following command to clear any version locks:

   yum versionlock clear

3. Run the following command to install the Leapp utility:

   yum install leapp leapp-repository

4. Download the the RedHat auxiliary files from RedHat for the upgrade. This is necessary for a successful OS upgrade.
1. Go to this RedHat knowledge base article: https://access.redhat.com/articles/3664871
2. Login using your RedHat credentials.
3. Scroll to the section Community Linux distributions and download the leapp data file. At the time of this writing, the latest version was leapp-data17.tar.gz.
5. SCP the following to each node in the UBA cluster:

   scp leapp-data17.tar.gz caspida@<uba-nodeX>:/home/caspida/

6. On the node, untar the leapp-data17.tar.gz package into the /etc/leapp/files directory.

   tar -xzf leapp-data17.tar.gz -C /etc/leapp/files && rm leapp-data17.tar.gz

7. On your RHEL 7 system, perform the pre-upgrade phase. Set the target flag to the desired RedHat version. 8.6:

   leapp preupgrade --target 8.6

8. After the preupgrade step, answerfile will be generated. Provide answers before proceeding with the upgrade. Use one of the following methods to answer the questions required by Leapp:
   • Give confirmation of the sections of the leapp answerfile by editing the /var/log/leapp/answerfile file and adding True after the statement confirm = for all the respective sections.
   • Use the leapp answer command to confirm all the sections individually using section names:

     leapp answer --section question_section.confirm=answer

     For example, to confirm a True response to the question Disable pam_pkcs11 module in PAM configuration?, execute the following command:

     leapp answer --section remove_pam_krb5_module_check.confirm=True

9. Examine the report in the /var/log/leapp/leapp-report.txtfile, and manually resolve all the reported problems before proceeding with the in-place upgrade.
10. During Leapp preupgrade you may encounter following inhibitors:

    Inhibitor	Solution
    Detected loaded kernel drivers which have been removed in RHEL 8. Upgrade cannot proceed.	Check the Leapp report for detailed instructions, to know which kernel driver is causing the error. Run the following command: rmmod pata_acpi
    Possible problems with remote login using root account	Uncomment line "PermitRootLogin" in /etc/ssh/sshd_config
    Newest installed kernel not in use.	Restart your machine.
    Missing required answers in the answer file	Provide answers to each section in the answerfile. See step 8.

    All other inhibitors can be safely ignored.

11. After solving all the inhibitors and giving answers to all the sections in the /var/log/leapp/answerfile file, repeat step 7 to verify that your system is ready for upgrade:
12. On your RHEL 7 system, start the upgrade process:

    leapp upgrade --target 8.6

    1. If the system is upgradeable, Leapp downloads the necessary data and prepares an RPM transaction for the upgrade.
    2. If the system does not meet the requirements for a reliable upgrade, Leapp terminates the upgrade process and provides a record describing the issue and a recommended solution in the /var/log/leapp/leapp-report.txt file.
13. Manually reboot the system:

    reboot

    1. In this phase, the system boots into a RHEL 8-based initial RAM disk image, initramfs. Leapp upgrades all packages and automatically reboots to the RHEL 8 system.
    2. Alternatively, you can run the leapp upgrade command with the --reboot option and skip this manual step.

System OS upgrade complete. The system is now at OS version 8.6.

Upgrade from RHEL 8.4 or 8.5 to RHEL 8.6

1. On each node, perform the OS upgrade steps as root user:

   subscription-manager release --set=8.6
   yum update
   reboot

2. Verify the OS version is upgraded to the desired version:

   cat /etc/os-release

System OS upgrade complete. The system is now at OS version 8.6.

Perform UBA upgrade

When upgrading the RHEL version from 7.9 or 8.x to 8.6 in a multi-node system, perform all the previous steps in this topic, in all the respective nodes, before completing the steps listed here.

Upgrade Splunk UBA 5.2.0 on the management node only. The upgrade script will update all relevant files on the other Splunk UBA nodes.

Perform the following tasks to upgrade Splunk UBA in a distributed environment.

1. Untar the UBA 5.2.0 Platform build inside the /opt/caspida/ folder:

   tar xvzf /home/caspida/Splunk-UBA-Platform-5.2.0-20230202-8254784.tgz -C /opt/caspida

2. Modify error conditions in the upgrade script:

   sed -i 's/--allowerasing -y >> \${LOG_FILE} 2>&1/--allowerasing -y 2>\&1 | tee \${LOG_FILE} | grep Error: \&\& exit 1/g' /opt/caspida/upgrade/utils/remove_install_os_packages.sh

3. Run the UBA upgrade script, using the path to your archived UBA from the Stop all UBA services and archive step:

   The path-to-prev-uba-archive might be /var/vcap/release_archives/caspida- depending on your archived UBA version number.

   /opt/caspida/upgrade/utils/upgrade_uba.sh -p /var/vcap/release_archives/caspida-<old-uba-version> -e /home/caspida/uba-ext-pkgs-5.2.0.tgz

   The command installs the new Splunk UBA software, restarts Splunk UBA, and then restarts the data sources.

If you have previously imported an output connector certificate, re-import the certificate. See, Configure the Splunk platform to receive data from the Splunk UBA output connector in the Send and Receive Data from the Splunk Platform manual.

Steps for 7 node deployments or larger

On 7 node deployments or larger, run the following steps:

1. SSH as caspida to UBA management node (node 1).
2. Back up the original ModelRegistry.json file:

   cp /opt/caspida/content/Splunk-Standard-Security/modelregistry/offlineworkflow/ModelRegistry.json 
   /opt/caspida/content/Splunk-Standard Security/modelregistry/offlineworkflow/ModelRegistry.json.ORIG

3. Copy the ModelRegistry.json.large_deployment file to the ModelRegistry.json file:

   cp /opt/caspida/content/Splunk-Standard-Security/modelregistry/offlineworkflow/ModelRegistry.json.large_deployment
   /opt/caspida/content/Splunk-Standard-Security/modelregistry/offlineworkflow/ModelRegistry.json

4. Sync cluster:

   /opt/caspida/bin/Caspida sync-cluster /opt/caspida/content/Splunk-Standard-Security/modelregistry/offlineworkflow/

5. Restart all services:

   /opt/caspida/bin/Caspida stop-all 
   /opt/caspida/bin/Caspida start-all

Re-import the certificate used for sending UBA audit events to Splunk ES

If you have enabled the integration that sends UBA audit data to Splunk Enterprise Security (ES), and you preserved the associated certificate in the Before you begin step, you can now re-import that certificate. Use the following command to re-import the certificate:

. /opt/caspida/bin/CaspidaCommonEnv.sh
sudo keytool -import -alias "splunk es" -keystore $JAVA_HOME/lib/security/cacerts -file ~/splunk-es_cacert.pem

Re-import the SSL certificate used for validating datasources from the Splunk Enterprise platform

If you have enabled the integration that sends UBA audit data to Splunk Enterprise Security (ES), use the following command to re-import the SSL certificate:

. /opt/caspida/bin/CaspidaCommonEnv.sh
sudo keytool -import -alias "SplunkESRootCA" -keystore $JAVA_HOME/lib/security/cacerts -file ~/SplunkESRootCA.pem

For more information on SSL certificate validation, see Configure flag to enable or disable Splunk SSL certificate validation.

Apply security patches on your Linux operating system on all nodes in the distributed deployment

Perform the following tasks to apply the latest Linux operating system security patches:

1. Log in to the Splunk UBA management node as the caspida user.
2. On the management node, run the following command to stop Splunk UBA and all services:

   /opt/caspida/bin/Caspida stop-all

3. On all nodes, run the following commands to check for any available security updates:

   sudo yum updateinfo list security all
   sudo yum updateinfo list sec
   

4. On all nodes, run the following command to update all packages with the available security updates:

   sudo yum update --security -y
   sudo yum --security update-minimal
   

5. On all nodes, reboot the system:

   sudo reboot

6. On the management node, run the following command to start Splunk UBA and all services:

   /opt/caspida/bin/Caspida start-all

Next steps

Verify a successful upgrade of Splunk UBA.