Upgrade a distributed OEL installation of Splunk UBA
Perform the following steps to upgrade a distributed OEL installation of Splunk UBA.
OEL Leapp regularly updates its base operating system to the latest version of OEL 8.x. If you are a Splunk UBA customer using OEL, refrain from upgrading OEL until you verify the OEL version Leapp will upgrade your environment to is supported by the target UBA version. For information on the latest OEL version, see https://docs.oracle.com/en/operating-systems/oracle-linux/8 /.
Before you begin
Make sure the correct hadoop ports are open. See, Inbound networking port requirements.
Before upgrading Splunk UBA in a distributed deployment, confirm you meet the Upgrade Splunk UBA prerequisites. Make sure that the prerequisites are verified on each server in the distributed deployment.
If you have enabled the integration that sends UBA audit data to Splunk Enterprise Security (ES), preserve the associated certificate before upgrading. For more info on sending audit data to ES see, Send Splunk UBA audit events to Splunk ES. Use the following command to export the certificate:
. /opt/caspida/bin/CaspidaCommonEnv.sh sudo keytool -exportcert -alias "splunk es" -keystore $JAVA_HOME/lib/security/cacerts -rfc -file ~/splunk-es_cacert.pem
If you have enabled validation of the SSL certificate from Splunk ES, export the SSL certificate used for validating datasources from the Splunk Enterprise platform:
. /opt/caspida/bin/CaspidaCommonEnv.sh sudo keytool -exportcert -alias "SplunkESRootCA" -keystore $JAVA_HOME/lib/security/cacerts -rfc -file ~/SplunkESRootCA.pem
Stop all UBA services and archive
Switch to the caspida user and perform the following steps to archive the older version of Splunk UBA:
- On the management node only, stop all the caspida services:
/opt/caspida/bin/Caspida stop-all
- On each node, archive the older version of UBA. REPLACE <old-uba-version> with your UBA version:
sudo mkdir -p /var/vcap/release_archives sudo mv /opt/caspida /var/vcap/release_archives/caspida-<old-uba-version> sudo mkdir -p /opt/caspida && sudo chown caspida:caspida /opt/caspida sudo chown caspida:caspida /var/vcap/release_archives sudo chown caspida:caspida /var/vcap/release_archives/caspida-<old-uba-version>
Upgrade your OEL operating system from 7.9 to 8.7
Perform the following tasks on each Splunk UBA node to upgrade your OEL operating system to version 8.7:
- Log in to the server as the root user.
- Perform the initial checks of the system.
- Set system locale to en_US.UTF-8 inside /etc/locale.conf file:
LANG="en_US.UTF-8" LC_CTYPE="en_US.UTF-8"
- Source the /etc/locale.conf file:
source /etc/locale.conf
- Clear any version locks:
sudo yum versionlock clear
- Obtain the latest Oracle Linux 7 packages:
sudo yum update -y
After the update completes, the following message might be displayed:
warning: /etc/yum.repos.d/oracle-linux-ol7.repo created as /etc/yum.repos.d/oracle-linux-ol7.repo.rpmnew
This warning appears if an oracle-linux-ol7.repo file already exists prior to updating the Oracle Linux 7 packages. The update process creates the .rpmnew file to avoid overwriting any customizations that might be in the current file.
In this case, use the .rpmnew file to guide you in making the necessary modifications to your existing .repo file. Incorporate any new information into your .repo file. The ol7_leapp repository description must be listed in your final oracle-linux-ol7.repo file for the upgrade to proceed. - Reboot the system:
sudo reboot
- Install the Leapp utility while enabling certain repositories, as follows:
sudo yum install -y leapp --enablerepo=ol7_leapp,ol7_latest
- Grant root SSH login permissions in the /etc/ssh/sshd_config file.
PermitRootLogin yes
- Run the preupgrade command.
sudo leapp preupgrade --oraclelinux
- After the preupgrade step completes, an answerfile is generated. Provide answers before proceeding with the upgrade. Provide answers to each question required by Leapp by either of the following methods:
- To give confirmation of the sections of the leapp answerfile, edit the - /var/log/leapp/answerfile file and add "True" after the statement "confirm = " for all the respective sections.
- Alternatively, use the leapp answer command to confirm all the sections individually using section names:
leapp answer --section question_section.confirm=answer
For example, to confirm a True response to the question Disable pam_pkcs11 module in PAM configuration?, execute the following command:
leapp answer --section remove_pam_krb5_module_check.confirm=True
The Leapp report might post the following:
Risk Factor: high Title: Difference in Python versions and support in OL 8 Summary: In OL 8, there is no 'python' command. Python 3 (backward incompatible) is the primary Python version and Python 2 is available with limited support an d limited set of packages. Read more here: https://docs.oracle.com/en/operating- systems/oracle-linux/8/python/ Remediation: [hint] Please run "alternatives --set python /usr/bin/python3" after upgrade
As the example shows, for some risks, the report suggests actions you should perform after the upgrade. Therefore, the risk, although high, is not labeled as an inhibitor. The remedy can be performed later.
- Run the preupgrade again.
- After solving all the inhibitors manually and giving answers of all the sections in the /var/log/leapp/answerfile file, repeat step 8 to check if the system is ready for upgrade.
-
Perform the upgrade:
sudo leapp upgrade --oraclelinux
- Manually reboot the system:
sudo reboot
- Update crypto policy:
sudo update-crypto-policies --set LEGACY sudo reboot
Upgrade your OEL operating system from 8.x to 8.7
Perform the following tasks on each Splunk UBA node to upgrade your OEL operating system to version 8.7:
- Log in to the server as the root user.
- Perform the initial check of the system.
- Set system locale to en_US.UTF-8 inside /etc/locale.conf file:
LANG="en_US.UTF-8" LC_CTYPE="en_US.UTF-8"
- Source the /etc/locale.conf file:
source /etc/locale.conf
- Set system locale to en_US.UTF-8 inside /etc/locale.conf file:
- Perform the upgrade:
sudo yum update -y
- Manually reboot the system:
reboot
Perform UBA upgrade
When upgrading the OEL version from 7.9 or 8.x to 8.7 in a multi-node system, perform all the previous steps in this topic, in all the respective nodes, before completing the steps listed here only on the management node.
Perform the following tasks to upgrade Splunk UBA in a distributed environment.
- Switch to caspida user and perform the following steps to archive the older version of UBA.
- On the management node, download the latest UBA branch build
splunk-uba-software-upgrade-package_520.tgz
and untar the UBA bits to/home/caspida directory
:$ tar xvzf /home/caspida/splunk-uba-software-upgrade-package_520.tgz Splunk-UBA-Platform-5.2.0-20230202-8254784.tgz Splunk-UBA-Platform-5.2.0-20230202-8254784.tgz.md5sum uba-ext-pkgs-5.2.0.tgz uba-ext-pkgs-5.2.0.tgz.md5sum
- Untar the UBA 5.2.0 Platform build inside the /opt/caspida/ folder:
tar xvzf /home/caspida/Splunk-UBA-Platform-5.2.0-20230202-8254784.tgz -C /opt/caspida
- Modify error conditions in the upgrade script:
sed -i 's/--allowerasing -y >> \${LOG_FILE} 2>&1/--allowerasing -y 2>\&1 | tee \${LOG_FILE} | grep Error: \&\& exit 1/g' /opt/caspida/upgrade/utils/remove_install_os_packages.sh
- Run the UBA upgrade script, using the path to your archived UBA from the Stop all UBA services and archive step.
The
path-to-prev-uba-archive
might be/var/vcap/release_archives/caspida-
depending on your archived UBA version number./opt/caspida/upgrade/utils/upgrade_uba.sh -p /var/vcap/release_archives/caspida-<old-uba-version> -e /home/caspida/uba-ext-pkgs-5.2.0.tgz
The command installs the new Splunk UBA software, restarts Splunk UBA, and then restarts the data sources.
If you have previously imported an output connector certificate, re-import the certificate. See, Configure the Splunk platform to receive data from the Splunk UBA output connector in the Send and Receive Data from the Splunk Platform manual.
Steps for 7 node deployments or larger
On 7 node deployments or larger, run the following steps:
- SSH as caspida to UBA management node (node 1).
- Back up the original ModelRegistry.json file:
cp /opt/caspida/content/Splunk-Standard-Security/modelregistry/offlineworkflow/ModelRegistry.json /opt/caspida/content/Splunk-Standard Security/modelregistry/offlineworkflow/ModelRegistry.json.ORIG
- Copy the ModelRegistry.json.large_deployment file to the ModelRegistry.json file:
cp /opt/caspida/content/Splunk-Standard-Security/modelregistry/offlineworkflow/ModelRegistry.json.large_deployment /opt/caspida/content/Splunk-Standard-Security/modelregistry/offlineworkflow/ModelRegistry.json
- Sync cluster:
/opt/caspida/bin/Caspida sync-cluster /opt/caspida/content/Splunk-Standard-Security/modelregistry/offlineworkflow/
- Restart all services:
/opt/caspida/bin/Caspida stop-all /opt/caspida/bin/Caspida start-all
Re-import the certificate used for sending UBA audit events to Splunk ES
If you have enabled the integration that sends UBA audit data to Splunk Enterprise Security (ES), and you preserved the associated certificate in the Before you begin step, you can now re-import that certificate. Use the following command to re-import the certificate:
. /opt/caspida/bin/CaspidaCommonEnv.sh sudo keytool -import -alias "splunk es" -keystore $JAVA_HOME/lib/security/cacerts -file ~/splunk-es_cacert.pem
Re-import the SSL certificate used for validating datasources from the Splunk Enterprise platform
If you have enabled the integration that sends UBA audit data to Splunk Enterprise Security (ES), use the following command to re-import the SSL certificate:
. /opt/caspida/bin/CaspidaCommonEnv.sh sudo keytool -import -alias "SplunkESRootCA" -keystore $JAVA_HOME/lib/security/cacerts -file ~/SplunkESRootCA.pem
For more information on SSL certificate validation, see Configure flag to enable or disable Splunk SSL certificate validation.
Apply security patches on your Linux operating system
Perform the following tasks to apply the latest Linux operating system security patches:
- Log in to the Splunk UBA server as the caspida user.
- Run the following command to stop Splunk UBA and all services:
/opt/caspida/bin/Caspida stop-all
- Run the following commands to check for any available security updates:
sudo yum updateinfo list security all sudo yum updateinfo list sec
- Run the following command to resolve glibc package dependencies:
sudo yum update glibc-devel
- Run the following command to update all packages with the available security updates:
sudo yum update --security -y sudo yum --security update-minimal
- Reboot the system:
sudo reboot
- Run the following command to start Splunk UBA and all services:
/opt/caspida/bin/Caspida start-all
Next steps
Upgrade a distributed RHEL installation of Splunk UBA | Upgrade a Splunk UBA deployment that is using warm standby |
This documentation applies to the following versions of Splunk® User Behavior Analytics: 5.2.0
Feedback submitted, thanks!