Upgrade a distributed AMI or OVA installation of Splunk UBA
Perform the following steps to upgrade a distributed AMI or OVA installation of Splunk UBA.
Upgrading to Splunk UBA version 5.2.0 on an AMI or OVA also requires you upgrade to Ubuntu version 20.04.
Before you begin
Make sure the correct hadoop ports are open. See, Inbound networking port requirements.
Before upgrading Splunk UBA in a distributed deployment, confirm you meet the Upgrade Splunk UBA prerequisites. Make sure that the prerequisites are verified on each server in the distributed deployment.
If you have enabled the integration that sends UBA audit data to Splunk Enterprise Security (ES), preserve the associated certificate before upgrading. For more info on sending audit data to ES see, Send Splunk UBA audit events to Splunk ES. Use the following command to export the certificate:
. /opt/caspida/bin/CaspidaCommonEnv.sh sudo keytool -exportcert -alias "splunk es" -keystore $JAVA_HOME/jre/lib/security/cacerts -rfc -file ~/splunk-es_cacert.pem
If you have enabled validation of the SSL certificate from Splunk ES, export the SSL certificate used for validating datasources from the Splunk Enterprise platform:
. /opt/caspida/bin/CaspidaCommonEnv.sh sudo keytool -exportcert -alias "SplunkESRootCA" -keystore $JAVA_HOME/jre/lib/security/cacerts -rfc -file ~/SplunkESRootCA.pem
Distributed environment upgrade steps
The Splunk UBA AMI and OVA images use Ubuntu as the operating system. Ensure that Splunk UBA is running before you upgrade, then perform the following tasks to upgrade Splunk UBA across multiple servers with the Ubuntu operating system.
To obtain and install Splunk UBA 5.2.0, perform the following tasks on the management node:
- On the management node only, stop all the caspida services.
/opt/caspida/bin/Caspida stop-all
- On each node, archive the previous UBA release. Replace
<old-uba-version>
with your version of UBA:sudo mkdir -p /var/vcap/release_archives sudo mv /opt/caspida /var/vcap/release_archives/caspida-<old-uba-version> sudo mkdir -p /opt/caspida && sudo chown caspida:caspida /opt/caspida sudo chown caspida:caspida /var/vcap/release_archives sudo chown caspida:caspida /var/vcap/release_archives/caspida-<old-uba-version>
.
- On each node, download the latest UBA branch build
splunk-uba-software-upgrade-package_520.tgz
and untar the UBA bits to/home/caspida
directory:$ tar xvzf /home/caspida/splunk-uba-software-upgrade-package_520.tgz Splunk-UBA-Platform-5.2.0-20230202-8254784.tgz Splunk-UBA-Platform-5.2.0-20230202-8254784.tgz.md5sum uba-ext-pkgs-5.2.0.tgz uba-ext-pkgs-5.2.0.tgz.md5sum
- On each node, untar the UBA 5.2.0 Platform build inside the /opt/caspida/ folder:
tar xvzf /home/caspida/Splunk-UBA-Platform-5.2.0-20230202-8254784.tgz -C /opt/caspida
- On each node update the package information.
- On each node, remove grub-pc package, then reinstall. This is done because there is an existing Ubuntu upgrade bug that is avoided with a clean install of this package.
- Remove by purging the grub-pc package:
sudo apt-get purge grub-pc -y
- Reinstall grub-pc package:
sudo apt-get install grub-pc -y
- When the interactive prompt comes up to choose a disk, select your BOOT disk if known. If BOOT disk is not known, select all disk choices.
Failure to choose the correct disk will result in it not being able to boot up.
- Remove by purging the grub-pc package:
- On each node, reboot the system:
sudo reboot
- If you are running Ubuntu 16.04, perform these steps:
- On each node, upgrade the OS from Ubuntu 16.04 to Ubuntu 18.04 with the following command:
sudo /opt/caspida/upgrade/utils/upgrade_ubuntu.sh -t 18.04
- On each node, after Ubuntu is upgraded, reboot the system:
sudo reboot
- Then upgrade from Ubuntu 18.04 to Ubuntu 20.04 on each node using these steps:
- On each node, install the newest available versions of all packages and reboot. Run the following commands:
sudo apt-get update sudo apt-get dist-upgrade sudo reboot
- On each node, upgrade the OS to Ubuntu 20.04 with the following command:
sudo /opt/caspida/upgrade/utils/upgrade_ubuntu.sh -t 20.04
- On each node, reboot the system:
sudo reboot
- On the management node, run the UBA upgrade script, using the path to your archived UBA from step 2. Migrate your configurations in a tmux shell just in case your SSH connection breaks.
The
path-to-prev-uba-archive
might be/var/vcap/release_archives/caspida-
depending on your archived UBA version number.$ tmux $ /opt/caspida/upgrade/utils/upgrade_uba.sh -p <path-to-prev-uba-archive> -e /home/caspida/uba-ext-pkgs-5.2.0.tgz
The command installs the new Splunk UBA software, restarts Splunk UBA, and then restarts the data sources.
sudo apt-get update
If you have previously imported an output connector certificate, re-import the certificate. See, Configure the Splunk platform to receive data from the Splunk UBA output connector in the Send and Receive Data from the Splunk Platform manual.
Steps for 7 node deployments or larger
On 7 node deployments or larger, run the following steps:
- SSH as caspida to UBA management node (node 1).
- Back up the original ModelRegistry.json file:
cp /opt/caspida/content/Splunk-Standard-Security/modelregistry/offlineworkflow/ModelRegistry.json /opt/caspida/content/Splunk-Standard Security/modelregistry/offlineworkflow/ModelRegistry.json.ORIG
- Copy the ModelRegistry.json.large_deployment file to the ModelRegistry.json file:
cp /opt/caspida/content/Splunk-Standard-Security/modelregistry/offlineworkflow/ModelRegistry.json.large_deployment /opt/caspida/content/Splunk-Standard-Security/modelregistry/offlineworkflow/ModelRegistry.json
- Sync cluster:
/opt/caspida/bin/Caspida sync-cluster /opt/caspida/content/Splunk-Standard-Security/modelregistry/offlineworkflow/
- Restart all services:
/opt/caspida/bin/Caspida stop-all /opt/caspida/bin/Caspida start-all
Re-import the certificate used for sending UBA audit events to Splunk ES
If you have enabled the integration that sends UBA audit data to Splunk Enterprise Security (ES), and you preserved the associated certificate in the Before you begin step, you can now re-import that certificate. Use the following command to re-import the certificate:
. /opt/caspida/bin/CaspidaCommonEnv.sh sudo keytool -import -alias "splunk es" -keystore $JAVA_HOME/jre/lib/security/cacerts -rfc -file ~/splunk-es_cacert.pem
Re-import the SSL certificate used for validating datasources from the Splunk Enterprise platform
If you have enabled the integration that sends UBA audit data to Splunk Enterprise Security (ES), use the following command to re-import the SSL certificate:
. /opt/caspida/bin/CaspidaCommonEnv.sh sudo keytool -import -alias "SplunkESRootCA" -keystore $JAVA_HOME/jre/lib/security/cacerts -file ~/SplunkESRootCA.pem
For more information on SSL certificate validation, see Configure flag to enable or disable Splunk SSL certificate validation.
Troubleshoot the Ubuntu upgrade
While upgrading Ubuntu some environment specific issues can arise. See the following known issues and how to work around them.
Snapd
You might encounter the following error message if snapd is in an unexpected state:
FileNotFoundError: [Errno 2] No such file or directory: 'snap'
Workaround steps:
- Remove and purge the snapd package:
sudo apt purge snapd
- Reinstall the snapd package:
sudo apt install snapd
- Re-run the
upgrade_ubuntu.sh
script.
Next steps
Upgrade a single node OEL installation of Splunk UBA | Upgrade a distributed RHEL installation of Splunk UBA |
This documentation applies to the following versions of Splunk® User Behavior Analytics: 5.2.0
Feedback submitted, thanks!