Splunk® User Behavior Analytics

Install and Upgrade Splunk User Behavior Analytics

This documentation does not apply to the most recent version of Splunk® User Behavior Analytics. For documentation on the most recent version, go to the latest release.

Upgrade a distributed AMI or OVA installation of Splunk UBA

Perform the following steps to upgrade a distributed AMI or OVA installation of Splunk UBA.

Upgrading to Splunk UBA version 5.4.0 on an AMI or OVA also requires you upgrade to Ubuntu version 20.04.

Prerequisites

Complete the following steps to ensure your system is correctly set up for upgrading Splunk UBA:

  1. Review the new features in this version of Splunk UBA for any changes that might need to be addressed before upgrading. See What's new in the Release Notes manual.
  2. Confirm you meet the Upgrade Splunk UBA prerequisites. Make sure that the prerequisites are verified on each server in the distributed deployment.
  3. Make sure the correct hadoop ports are open. See, Inbound networking port requirements.
  4. Ensure you Verify Postgres Collate and Ctype values.
  5. If you have enabled the integration that sends UBA audit data to Splunk Enterprise Security (ES), preserve the associated certificate before upgrading.
    Use the following command to export the certificate: 
    . /opt/caspida/bin/CaspidaCommonEnv.sh
sudo keytool -exportcert -alias "splunk es" -keystore $JAVA_HOME/lib/security/cacerts -rfc -file ~/splunk-es_cacert.pem
    For more info on sending audit data to ES see, Send Splunk UBA audit events to Splunk ES in the Send and Receive Data from the Splunk Platform manual.
  6. If you have enabled validation of the SSL certificate from Splunk ES, export the SSL certificate used for validating datasources from the Splunk Enterprise platform: 
    . /opt/caspida/bin/CaspidaCommonEnv.sh
sudo keytool -exportcert -alias "SplunkESRootCA" -keystore $JAVA_HOME/lib/security/cacerts -rfc -file ~/SplunkESRootCA.pem
  7. If you have enabled the integration that sends UBA events to Splunk ES, add connection_host = ip to the HTTP Event Collector (HEC) inputs.conf on the ES search head.
    For example: 
     /opt/splunk/etc/apps/splunk_httpinput/local/inputs.conf
    This ensures that the host field remains the sender's (UBA) IP address instead of the default HEC host and port.
  8. Customers with existing UBA-ES integrations must comment out or remove the previously configured [tcp-ssl:10008] stanza from the Splunk_TA_ueba inputs.conf on the Splunk ES search head to avoid having an unused listener.

Distributed environment upgrade steps

The Splunk UBA AMI and OVA images use Ubuntu as the operating system. Ensure that Splunk UBA is running before you upgrade, then perform the following tasks to upgrade Splunk UBA across multiple servers with the Ubuntu operating system.

To obtain and install Splunk UBA 5.4.0, perform the following tasks on the management node:

  1. On the management node only, stop all the caspida services. 
    /opt/caspida/bin/Caspida stop-all
  2. On each node, archive the previous UBA release. Replace <old-uba-version> with your version of UBA: 
    sudo mkdir -p /var/vcap/release_archives
sudo mv /opt/caspida /var/vcap/release_archives/caspida-<old-uba-version>
sudo mkdir -p /opt/caspida && sudo chown caspida:caspida /opt/caspida
sudo chown caspida:caspida /var/vcap/release_archives
sudo chown caspida:caspida /var/vcap/release_archives/caspida-<old-uba-version>
  3. On each node, download the latest UBA branch build splunk-uba-software-upgrade-package_540.tgz and untar the UBA bits to /home/caspida directory: 
    $ tar xvzf /home/caspida/splunk-uba-software-upgrade-package_540.tgz
Splunk-UBA-Platform-5.4.0-20240424-16474780.tgz
Splunk-UBA-Platform-5.4.0-20240424-16474780.tgz.md5sum
uba-ext-pkgs-5.4.0.tgz
uba-ext-pkgs-5.4.0.tgz.md5sum
  4. On each node, untar the UBA 5.4.0 Platform build inside the /opt/caspida/ folder: 
    tar xvzf /home/caspida/Splunk-UBA-Platform-5.4.0-20240424-16474780.tgz -C /opt/caspida

    5. If you are already using Ubuntu 20.04, there is no need to upgrade Ubuntu. Proceed directly to step 9.

  5. On each node update the package information.
    6. sudo apt-get update
  6. On each node, remove grub-pc package, then reinstall. This is done because there is an existing Ubuntu upgrade bug that is avoided with a clean install of this package.
    1. Remove by purging the grub-pc package: 
      sudo apt-get purge grub-pc -y
    2. Reinstall grub-pc package: 
      sudo apt-get install grub-pc -y
    3. When the interactive prompt comes up to choose a disk, select your BOOT disk if known. If BOOT disk is not known, select all disk choices.

      Failure to choose the correct disk will result in it not being able to boot up.

  7. On each node, reboot the system: 
    sudo reboot
  8. Upgrade from Ubuntu 18.04 to Ubuntu 20.04 on each node using these steps:
    1. On each node, install the newest available versions of all packages and reboot. Run the following commands: 
      sudo apt-get update
sudo apt-get dist-upgrade
sudo reboot
    2. On each node, upgrade the OS to Ubuntu 20.04 with the following command: 
      sudo /opt/caspida/upgrade/utils/upgrade_ubuntu.sh -t 20.04
    3. On each node, reboot the system: 
      sudo reboot
  9. Set or update the following environment variable for PostgreSQL in the /etc/default/locale file:
    Change the LANG value to en_US.UTF-8
  10. Run the following command to source /etc/default/locale:
    source /etc/default/locale
  11. (Optional) Follow the steps to turn on FIPS compliance. See Turn on FIPS compliance.
  12. On the management node, run the UBA upgrade script, using the path to your archived UBA from step 2. Migrate your configurations in a tmux shell just in case your SSH connection breaks.

    The path-to-prev-uba-archive might be /var/vcap/release_archives/caspida- depending on your archived UBA version number.

    $ tmux
$ /opt/caspida/upgrade/utils/upgrade_uba.sh -p <path-to-prev-uba-archive> -e /home/caspida/uba-ext-pkgs-5.4.0.tgz
    The command installs the new Splunk UBA software, restarts Splunk UBA, and then restarts the data sources.
  13. Ensure the proper configuration of the Splunk forwarder. Commands must be executed on the management node.
    1. Stop UBA's Splunk forwarder: 
      /opt/caspida/bin/Caspida stop-splunk
    2. Run the following command to update TCP connector properties names in outputs.conf to the correct format: 
      sed -i \
 -e 's/sslpassword/sslPassword/g' \
 -e 's/autolbfrequency/autoLBFrequency/g' \
 -e 's/autolbvolume/autoLBVolume/g' \
 -e 's/forcetimebasedautolb/forceTimebasedAutoLB/g' \
 -e 's/clientcert/clientCert/g' \
 -e 's/sslrootcapath/sslRootCAPath/g' \
 -e 's/sslcommonnametocheck/sslCommonNameToCheck/g' \
 -e 's/sslverifyservercert/sslVerifyServerCert/g' \
 -e 's/useclientsslcompression/useClientSSLCompression/g' \
 /opt/splunk/etc/apps/Splunk_UBA_Monitor/default/outputs.conf
    3. Clean fishbucket to force re-indexing of the telemetry.data.out events file: 
      [ -f /var/log/caspida/telemetry/events/telemetry.data.out ] && /opt/splunk/bin/splunk cmd btprobe -d /opt/splunk/var/lib/splunk/fishbucket/splunk_private_db --file /var/log/caspida/telemetry/events/telemetry.data.out --reset
    4. Start UBA's Splunk forwarder: 
      /opt/caspida/bin/Caspida start-splunk

If you have previously imported an output connector certificate, re-import the certificate. See, Configure the Splunk platform to receive data from the Splunk UBA output connector in the Send and Receive Data from the Splunk Platform manual.

Turn on FIPS compliance

Turning on FIPS compliance is an optional step when installing or upgrading Splunk UBA.

Federal Information Processing Standard (FIPS) compliance is available with Splunk UBA version 5.4.0 and higher. Complete the following steps to turn on FIPS on each Splunk UBA node before running the upgrade script.

For developing and running workloads with FIPS on the enterprise, the validated packages are available with Ubuntu Pro or an Ubuntu Advantage subscription. See https://ubuntu.com/public-cloud and https://ubuntu.com/advantage .

Perform the following steps on each node:

  1. Update the package information: 
    sudo apt update
  2. Install ubuntu-advantage-tools in your Ubuntu system: 
    sudo apt install ubuntu-advantage-tools
  3. Sign up for Ubuntu Pro using the following website: https://ubuntu.com/advantage
  4. Once logged in, go to the Ubuntu Pro Dashboard and copy the token.
  5. Replace the copied token with <Token> and run the following command: 
    sudo ua attach <TOKEN>
  6. Turn on FIPS using the fips-updates stream on Ubuntu LTS.: 
    sudo ua enable fips-updates
  7. After successfully turning on FIPS, reboot the system: 
    sudo reboot
  8. Check the status to confirm FIPS is turned on: 
    sudo ua status
  9. You can also verify the status using the following command:

    You see a 1 if FIPS is turned on, otherwise 0.

    cat /proc/sys/crypto/fips_enabled

To learn more about FIPS in Ubuntu systems, see https://ubuntu.com/blog/running-fips-140-workloads-on-ubuntu.

Apply security patches on Ubuntu

The Splunk UBA AMI and OVA images use Ubuntu as the operating system. Perform the following tasks to apply the latest security patches to your Ubuntu operating system:

Applying the security patches can take up to one hour.

  1. Log in to the Splunk UBA management node as the caspida user.
  2. On the management node, run the following command to stop Splunk UBA and all services: 
    /opt/caspida/bin/Caspida stop-all
  3. On each Splunk UBA node, perform the following tasks:
    1. Log in as the caspida user.
    2. Run the following commands to install latest unattended-upgrades package: 
      sudo apt update
sudo apt install unattended-upgrades

      If you see the following prompt, select keep the local version currently installed: 

      What do you want to do about modified configuration file 50unattended-upgrades?
    3. Run the following command: 
      sudo apt autoremove
    4. Edit the /etc/apt/apt.conf.d/50unattended-upgrades file and un-comment the following line:

      Skip this step if you have previously applied security patches to your Ubuntu environment following these instructions.

      "${distro_id}:${distro_codename}-security";
      Leave all other lines commented out.
    5. Run the following command: 
      sudo unattended-upgrade -d
    6. Edit the /etc/init.d/zookeeper-server file and change su to runuser in all of the following lines:

      Skip this step if you have previously applied security patches to your Ubuntu environment following these instructions.

      Before: 

      su -s /bin/bash zookeeper -c "${DAEMON_SCRIPT} start"
su -s /bin/bash zookeeper -c "${DAEMON_SCRIPT} stop"
su -s /bin/bash zookeeper -c "zookeeper-server-initialize $*"

      After: 

      runuser -s /bin/bash zookeeper -c "${DAEMON_SCRIPT} start"
runuser -s /bin/bash zookeeper -c "${DAEMON_SCRIPT} stop"
runuser -s /bin/bash zookeeper -c "zookeeper-server-initialize $*"
    7. Reboot the system: 
      sudo reboot
  4. On the management node, run the following command to start Splunk UBA and all services: 
    /opt/caspida/bin/Caspida start-all

Re-import the certificate used for sending UBA audit events to Splunk ES

If you have enabled the integration that sends UBA audit data to Splunk Enterprise Security (ES), and you preserved the associated certificate in the Before you begin step, you can now re-import that certificate. Use the following command to re-import the certificate: 

. /opt/caspida/bin/CaspidaCommonEnv.sh
sudo keytool -import -alias "splunk es" -keystore $JAVA_HOME/jre/lib/security/cacerts -rfc -file ~/splunk-es_cacert.pem

Re-import the SSL certificate used for validating datasources from the Splunk Enterprise platform

If you have enabled the integration that sends UBA audit data to Splunk Enterprise Security (ES), use the following command to re-import the SSL certificate: 

. /opt/caspida/bin/CaspidaCommonEnv.sh
sudo keytool -import -alias "SplunkESRootCA" -keystore $JAVA_HOME/jre/lib/security/cacerts -file ~/SplunkESRootCA.pem

For more information on SSL certificate validation, see Configure flag to enable or disable Splunk SSL certificate validation.

Troubleshoot the Ubuntu upgrade

While upgrading Ubuntu some environment specific issues can arise. See the following known issues and how to work around them.

Snapd

You might encounter the following error message if snapd is in an unexpected state:

FileNotFoundError: [Errno 2] No such file or directory: 'snap'

Workaround steps:

  1. Remove and purge the snapd package:
    sudo apt purge snapd
  2. Reinstall the snapd package:
    sudo apt install snapd
  3. Re-run the upgrade_ubuntu.sh script.

Next steps

You can Verify a successful upgrade of Splunk UBA.

By default, the caspida user is given ALL access in /etc/sudoers during Splunk UBA installation and upgrade. If you want to restrict sudo access for the caspida user after Splunk UBA is upgraded, see Restrict sudo access for the caspida account.

Last modified on 27 August, 2024
Upgrade a single node OEL installation of Splunk UBA   Upgrade a distributed RHEL installation of Splunk UBA

This documentation applies to the following versions of Splunk® User Behavior Analytics: 5.4.0

Acrobat logo Download manual
Acrobat logo Download this page

Was this topic useful?







You must be logged into splunk.com in order to post comments. Log in now.

Please try to keep this discussion focused on the content covered in this documentation topic. If you have a more general question about Splunk functionality or are experiencing a difficulty with Splunk, consider posting a question to Splunkbase Answers.

0 out of 1000 Characters