Install Splunk UBA on several Linux servers

Install Splunk UBA on several servers with Oracle Enterprise Linux (OEL) or Red Hat Enterprise Linux (RHEL) installed. Splunk UBA 5.2.0 requires OEL 8.7 or RHEL 8.6.

Follow these instructions to install Splunk UBA 5.2.0 for the first time. If you already have Splunk UBA, do not follow the instructions on this page. Instead, follow the appropriate upgrade instructions to obtain your desired release. See, How to install or upgrade to this release of Splunk UBA.

Prerequisites for installing Splunk UBA on several Linux servers

• You must install Splunk UBA on a server that is running a supported operating system. See, Operating system requirements.
• Make sure your Red Hat Enterprise Linux license includes the proper subscription names. See, Additional RHEL requirements.
• Determine the interface of your system network configuration, for example eth0 or en0. You will need this information later in the installation process.
• The bind-utils package must be installed on your system. Use the following command to install:

yum install bind-utils

Configure permissions for and prepare the caspida user

Enable sudo permissions for the caspida user.

1. Use the visudo command to edit the /etc/sudoers file.
2. If the following line exists, comment the line Defaults requiretty.
3. Add the following lines at the end of the /etc/sudoers file.

   caspida ALL=(ALL) NOPASSWD:ALL
   Defaults secure_path = /sbin:/bin:/usr/sbin:/usr/bin
   

   The /etc/sudoers file is read sequentially, so placing these lines at the end ensures that there is no impact to the caspida user from any existing accounts or group permissions.
4. Add the caspida user to the systems. The caspida user needs to have the same UID and GID on all the systems. Pick a UID and GID that is available on all the systems. For example, assuming UID and GID 2018 is available on all nodes:

   groupadd --gid 2018 caspida
   useradd --uid 2018 --gid 2018 -m -d /home/caspida -c "Caspida User" -s /bin/bash caspida
   

5. Set the password for caspida user:

   passwd caspida
   

6. Verify the caspida user permissions for newly created files and directories. See Validate the UMASK value.

Obtain the installation package

Download the following Splunk UBA software and RHEL packages.

1. Obtain the Splunk UBA 5.2.0 software:
   1. Go to the Splunk UBA Software Installation Package page on Splunkbase.
   2. Download the file to the /home/caspida directory. The name of the package is splunk-uba-software-installation-package_520.tgz.

Use these packages for all supported Linux environments.

Prepare all servers in your distributed Linux environment

Perform these steps on every server node in the distributed deployment.

1. From the command line, log in to the server as the root user, or log in as a different user then use su or sudo to gain root user privileges.
2. Double-check your hostnames prior to running the setup. Verify the hostname of all the nodes using the following command:

   hostname

   1. Make sure all the nodes have a consistent hostname setup. If using fully qualified domain names (FQDN) then all nodes should output FQDN in the hostname command. If the short name is used, then all nodes should output the short name in the hostname command.
3. Make sure all the nodes have a consistent setup. If using fully qualified domain names (FQDN) then all nodes should output FQDN in the hostname command. If the short name is used, then all nodes should output the short name in the hostname command.

   If you plan on connecting to Splunk Cloud to run queries for datasources, use fully qualified domain names (FQDN), not short names, for your Splunk UBA hostnames.

4. Find the additional 1TB disk or disks using the fdisk -l command. The nodes that are running Spark services should have two 1TB disks. See Disk space and memory requirements for a summary of where Spark is running per deployment.
   For example, you may see two disks named /dev/sdb and /dev/sdc.
5. Partition and format the partition on each disk found in step 2.
   1. Partition and format the partition on the /dev/sdb disk using the following series of commands. Verify that the align-check opt 1 command returns 1 aligned.

      parted -a optimal /dev/sdb
        mklabel gpt
        mkpart primary ext4 2048s 100%
        align-check opt 1
        quit
      

   2. Format the partition using the mkfs command.

      mkfs -t ext4 /dev/sdb1

   3. On nodes that have a second disk, repeat the commands to partition and format the partition on /dev/sdc:

      parted -a optimal /dev/sdc
        mklabel gpt
        mkpart primary ext4 2048s 100%
        align-check opt 1
        quit
      

   4. Format the partition using the mkfs command. When prompted, confirm that you want to continue.

      mkfs -t ext4 /dev/sdc1

6. Get the block ID for each disk using the blkid command. For example, to get the block IDs for /dev/sdb1 and /dev/sdc1 in our example:

   blkid -o value -s UUID /dev/sdb1
   blkid -o value -s UUID /dev/sdc1
   

   An example block ID might be: 5c00b211-e751-4661-91c4-60d9f9315857.
7. Create new /var/vcap and /var/vcap2 directories. For example, on a node with a single 1TB disk:

   mkdir -p /var/vcap

   Or on a node with two 1TB disks:

   mkdir -p /var/vcap /var/vcap2

8. Add the block ID for the /var/vcap partition to the /etc/fstab directory. For example, on a node with a single 1TB disk:

   UUID=e1af8814-9b12-4c69-a947-18af370c7dd1 /var/vcap  ext4  defaults  0 0
   

   On a node with two 1TB disks:

   UUID=e1af8814-9b12-4c69-a947-18af370c7dd1 /var/vcap  ext4  defaults  0 0
   UUID=f142f182-27c6-4002-b0bb-941fbedce17d /var/vcap2  ext4  defaults  0 0
   

9. Mount the file systems using the mount -a command.
10. Verify that the 1TB disks are mounted correctly using the df -h command. For example:

    root# df -h
    Filesystem      Size  Used Avail Use% Mounted on
    ...
    /dev/sdc1       493G   77M  467G   1% /var/vcap2
    /dev/sdb1       985G   43G  892G   5% /var/vcap
    ...
    

11. Inherit the permissions for the root user. On a node with a single 1TB disk:

    chmod 755 /var/vcap
    chown root:root /var/vcap
    

    Or on a node with two 1TB disks:

    chmod 755 /var/vcap /var/vcap2
    chown root:root /var/vcap /var/vcap2
    

12. Make a directory for caspida software packages.

    This should be different from caspida home directory (/home/caspida).

    mkdir /opt/caspida
    chown caspida:caspida /opt/caspida
    chmod 755 /opt/caspida

13. Set the following environment variables for PostgreSQL in the /etc/locale.conf file:

    LANG="en_US.UTF-8"
    LC_CTYPE="en_US.UTF-8"
    

14. Run the following command to source the /etc/locale.conf file:

    source /etc/locale.conf

15. Verify that the host name resolves using the nslookup <host name> command. If it does not, verify your host name lookup and DNS settings. See Configure host name lookups and DNS.
16. Verify that the system date, time and time zone are correct using the timedatectl command, as shown here. The time zone in Splunk UBA should match the time zone configured in Splunk Enterprise.

    root# timedatectl status
          Local time: Mon 2019-04-08 14:30:02 UTC
      Universal time: Mon 2019-04-08 14:30:02 UTC
            RTC time: Mon 2019-04-08 14:30:01
           Time zone: UTC (UTC, +0000)
         NTP enabled: yes
    NTP synchronized: yes
     RTC in local TZ: no
          DST active: n/a
    

    Use the timedatectl command to change the time zone. For example, to change the time zone to UTC:

    timedatectl set-timezone UTC

    Refer to the documentation for your specific operating system to configure NTP synchronization. Use the ntpq -p command to verify that NTP is pointing to the desired server.
17. Modify /etc/sysconfig/selinux and set SELINUX=permissive.
    With SELINUX set to enforced, certain actions during installation and upgrade (for example, access to particular files) can be blocked. Set SELINUX to permissive to allow Splunk UBA the necessary access so that actions are not blocked, but instead logged in the audit logs.
18. Verify that /proc/sys/net/bridge/bridge-nf-call-iptables exists on your system and the content of bridge-nf-call-iptables is 1. Run the following command to verify:

    cat /proc/sys/net/bridge/bridge-nf-call-iptables

    Your situation	Take this action
    /proc/sys/net/bridge/bridge-nf-call-iptables exists on your system and the content is 1.	Run the following command to make sure this setting is preserved through any reboot operations: echo net.bridge.bridge-nf-call-iptables=1 > /etc/sysctl.d/splunkuba-bridge.conf Go to Step 19.
    /proc/sys/net/bridge/bridge-nf-call-iptables exists on your system but the content is not 1.	Run the following commands to set the content of the bridge-nf-call-iptables: sysctl -w net.bridge.bridge-nf-call-iptables=1 Run the following command to ensure that the settings persist through any reboot operations: echo net.bridge.bridge-nf-call-iptables=1 > /etc/sysctl.d/splunkuba-bridge.conf Go to Step 19.
    /proc/sys/net/bridge/bridge-nf-call-iptables does not exist on your system.	Run the following commands to create the file and ensure that it is loaded on reboot: modprobe br_netfilter echo br_netfilter > /etc/modules-load.d/br_netfilter.conf Run the following commands to create and set the content of the bridge-nf-call-iptables: sysctl -w net.bridge.bridge-nf-call-iptables=1 Run the following command to ensure that the settings persist through any reboot operations: echo net.bridge.bridge-nf-call-iptables=1 > /etc/sysctl.d/splunkuba-bridge.conf Go to Step 19.

19. Run the following command to ensure that /etc/sysctl.d/splunkuba-bridge.conf is readable by the caspida user:

    chmod o+r /etc/sysctl.d/splunkuba-bridge.conf 

    If that command returns without error, proceed to the next step.

20. Verify that IPv6 drivers are available. To do this, check that /proc/sys/net/ipv6/ exists. For example:

    root# ls -l /proc/sys/net/ipv6/
    total 0
    -rw-r--r-- 1 root root 0 Mar 12 16:52 anycast_src_echo_reply
    -rw-r--r-- 1 root root 0 Mar 12 16:52 auto_flowlabels
    -rw-r--r-- 1 root root 0 Mar 12 16:52 bindv6only
    dr-xr-xr-x 1 root root 0 Mar 12 16:52 conf
    -rw-r--r-- 1 root root 0 Mar 12 16:52 flowlabel_consistency
    -rw-r--r-- 1 root root 0 Mar 12 16:52 flowlabel_state_ranges
    -rw-r--r-- 1 root root 0 Mar 12 16:52 fwmark_reflect
    dr-xr-xr-x 1 root root 0 Mar 12 16:52 icmp
    -rw-r--r-- 1 root root 0 Mar 12 16:52 idgen_delay
    -rw-r--r-- 1 root root 0 Mar 12 16:52 idgen_retries
    -rw-r--r-- 1 root root 0 Mar 12 16:52 ip6frag_high_thresh
    -rw-r--r-- 1 root root 0 Mar 12 16:52 ip6frag_low_thresh
    -rw-r--r-- 1 root root 0 Mar 12 16:52 ip6frag_secret_interval
    -rw-r--r-- 1 root root 0 Mar 12 16:52 ip6frag_time
    -rw-r--r-- 1 root root 0 Mar 12 16:52 ip_nonlocal_bind
    -rw-r--r-- 1 root root 0 Mar 12 16:52 mld_max_msf
    -rw-r--r-- 1 root root 0 Mar 12 16:52 mld_qrv
    dr-xr-xr-x 1 root root 0 Mar 12 16:52 neigh
    dr-xr-xr-x 1 root root 0 Mar 12 16:52 route
    -rw-r--r-- 1 root root 0 Mar 12 16:52 xfrm6_gc_thresh
    

    If the IPv6 drivers exist, skip to the next step.

    
    If IPv6 drivers do not exist on your system, check if /etc/default/grub contains ipv6.disable=1. IPv6 drivers will not be available on a system if ipv6.disable=1 exists in /etc/default/grub. If ipv6.disable=1 is not present in /etc/default/grub and IPv6 drivers do not exist, consult with your system or network administrators. You will not be able to continue with the installation.

    
    If /etc/default/grub contains ipv6.disable=1, perform the following tasks as root:

    1. Remove ipv6.disable=1 from /etc/default/grub.
    2. Recreate the grub config:

       find /boot -name grub.cfg -exec grub2-mkconfig -o '{}' \;

    3. Reboot the machines. After the system comes up, make sure /proc/sys/net/ipv6 exists.

    To disable IPv6 functionality for security, networking or performance reasons, create the /etc/sysctl.d/splunkuba-ipv6.conf file as root. This file should contain the following content:

    net.ipv6.conf.all.disable_ipv6 = 1
    net.ipv6.conf.default.disable_ipv6 = 1
    net.ipv6.conf.lo.disable_ipv6 = 1
    

    This procedure keeps the IPv6 drivers but disables the IPv6 addressing.
21. Create the /etc/security/limits.d/caspida.conf file and add the following security limits for the caspida user to this file:

    caspida soft nproc unlimited
    caspida soft nofile 32768
    caspida hard nofile 32768
    caspida soft core unlimited
    caspida soft stack unlimited
    caspida soft memlock unlimited
    caspida hard memlock unlimited
    

    Make sure the root account does not have any security limits.

22. If you are not using IPv6 on your network, edit the /etc/yum.conf file and add the following entry so that only IPv4 addresses are used by yum/rpm:

    ip_resolve=4

23. If you have any firewall configuration enabled, disable the configuration and verify that port 9002 is open. Run the following command:

    systemctl disable firewalld

    You can re-enable your firewall settings after the setup is complete.
24. Restart the system.

    init 6

25. After the system restarts, use the following command to verify that the host name matches your host name lookup and DNS settings. See Configure host name lookups and DNS.

    hostname --fqdn

Install Splunk UBA on each Linux server

Perform these steps on every server node in the distributed deployment to install Splunk UBA. If you are running the commands from the /home/caspida directory, you can omit the /home/caspida portion of the commands.

1. Log in to the command line as the caspida user using SSH.
2. Verify that the caspida user has umask permissions set to 0022 or 0002.
   umask
   If the returned values are not supported, edit the ~/.bash_profile and the ~/.bashrc files and append umask 0022
3. Verify that the splunk-uba-software-installation-package_520.tgz file that you downloaded in Obtain the installation package is in the /home/caspida directory. If not, copy the files to the /home/caspida directory.
4. Untar the file for Splunk UBA Software Installation in the /home/caspida directory.

   tar xvzf /home/caspida/splunk-uba-software-installation-package_520.tgz

5. Run the following command to untar the Splunk UBA platform software to the /opt/caspida directory:

   tar xvzf /home/caspida/Splunk-UBA-Platform-5.2.0-20230202-8254784.tgz -C /opt/caspida/

6. Run the following command to untar the Splunk UBA packages to the /home/caspida directory:

   tar xvzf /home/caspida/Splunk-UBA-5.2-Packages-RHEL-8.tgz -C /home/caspida

7. Run the installation script.

   /opt/caspida/bin/installer/redhat/INSTALL.sh /home/caspida/Splunk-UBA-5.2-Packages-RHEL-8
   

   The log file is /var/log/caspida/install.log.
1. If you see the following error, your OS has a newer version of the library krb5-libs and/or zlib:

   krb5-libs(x86-64) = 1.18.2-21.0.1.el8 is needed by krb5-devel-1.18.2-21.0.1.el8.x86_64
   krb5-libs(x86-64) = 1.18.2-21.0.1.el8 is needed by libkadm5-1.18.2-21.0.1.el8.x86_64
   zlib(x86-64) = 1.2.11-20.el8 is needed by zlib-devel-1.2.11-20.el8.x86_64 on Redhat enterprise linux
   

2. Fix this error by running the following commands to manually install:

   sudo yum -y localinstall /home/caspida/Splunk-UBA-5.2-Packages-RHEL-8/extra_packages/rpm/hadoop/krb5-libs-1.18.2-21.0.1.el8.x86_64.rpm
   sudo yum -y localinstall /home/caspida/Splunk-UBA-5.2-Packages-RHEL-8/extra_packages/rpm/hadoop/zlib-1.2.11-20.el8.x86_64.rpm
   

3. Then re-run the installation script
8. Generate SSH keys using the ssh-keygen -t rsa command. Press enter for all the prompts and accept all default values. For example:

   [caspida@ubahost-001]$ ssh-keygen -t rsa
   Generating public/private rsa key pair.
   Enter file in which to save the key (/home/caspida/.ssh/id_rsa):
   Created directory '/home/caspida/.ssh'.
   Enter passphrase (empty for no passphrase):
   Enter same passphrase again:
   Your identification has been saved in /home/caspida/.ssh/id_rsa.
   Your public key has been saved in /home/caspida/.ssh/id_rsa.pub.
   The key fingerprint is:
   SHA256:Ohe1oSpUtNT8siJzvn2lFLrHmVH7JGKke+c/5NRFb/g caspida@ubahost-001
   

9. Add the SSH keys to the server and adjust the permissions to allow the server to access them.
   

   cat /home/caspida/.ssh/id_rsa.pub >> /home/caspida/.ssh/authorized_keys
   chmod 600 /home/caspida/.ssh/authorized_keys

10. Copy the SSH keys from every server from /home/caspida/.ssh/id_rsa.pub into the /home/caspida/.ssh/authorized_keys file on every server in the distributed deployment. After this is complete, each authorized_keys file on each server should have all the SSH keys for every server in the deployment listed.
11. Test to verify that the SSH connections do not require a password. Connect to each server without a password with the host name using SSH to create trusted connections between the servers. After you confirm the connection does not require a password, use exit to terminate the SSH connection. You must complete this step before continuing with setup.

    ssh <node1>; exit
    ssh <node2>; exit
    ssh <node3>; exit

12. When prompted, confirm that you want to continue. For example, the sample output will look similar to the following.
    

    [caspida@ubahost-001]$ ssh uba1
    The authenticity of host 'uba1 (172.31.35.204)' can't be established.
    ECDSA key fingerprint is af:12:54:60:f5:36:c2:36:9d:56:b2:52:9f:cb:73:bc.
    Are you sure you want to continue connecting (yes/no)? yes
    Warning: Permanently added 'uba1,172.31.35.204' (ECDSA) to the list of known hosts.
    

Complete the distributed Linux Splunk UBA installation on the management server

After completing the previous steps on all server nodes in the distributed deployment, perform the following steps on the management server. For example, uba-node01.

1. Check the system status with the uba_pre_check.sh shell script.

   If you used FQDN instead of short-names to setup UBA, use FQDNs when running the uba_pre_check.sh script.

   You must specify the host name of each Splunk UBA node in the command, separated by spaces. For example, run the following command in a 3-node deployment and be sure to replace <node1> <node2> <node3> with the actual host names of your Splunk UBA nodes.

   /opt/caspida/bin/utils/uba_pre_check.sh <node1> <node2> <node3>

   See Check system status before and after installation for more information about the script.

2. Run the setup script.

   /opt/caspida/bin/Caspida setup

   1. When prompted, accept the license agreement and confirm removal of existing metadata.
   2. When prompted, type a comma-separated list of host names for your distributed installation. For example, specify the following in a 3-node deployment and be sure to replace <node1> <node2> <node3> with the actual host names of your Splunk UBA nodes:

      <node1>,<node2>,<node3>

   3. When prompted for a list of host names in the setup script, if the output of the host name command is FQDN, then provide a CSV list of FQDN host names, for example:

      <NODE1_FQDN>,<NODE2_FQDN>,<NODE3_FQDN>

   4. When prompted, confirm that you want to continue setting up Splunk UBA.
   5. The log file is /var/log/caspida/caspida.out
3. Perform one final sync across your cluster:

   /opt/caspida/bin/Caspida sync-cluster

4. Restart UBA:

   /opt/caspida/bin/Caspida stop
   /opt/caspida/bin/Caspida start

5. On 7 node deployments or larger, run the following steps:
   1. SSH as caspida to UBA management node (node 1).
   2. Back up original file:

      cp /opt/caspida/content/Splunk-Standard-Security/modelregistry/offlineworkflow/ModelRegistry.json 
      /opt/caspida/content/Splunk-Standard Security/modelregistry/offlineworkflow/ModelRegistry.json.ORIG 

   3. Copy the ModelRegistry.json.large_deployment file to the ModelRegistry.json file:

      cp /opt/caspida/content/Splunk-Standard-Security/modelregistry/offlineworkflow/ModelRegistry.json.large_deployment /opt/caspida/content/Splunk-Standard-Security/modelregistry/offlineworkflow/ModelRegistry.json
      

   4. Sync cluster:

      /opt/caspida/bin/Caspida sync-cluster /opt/caspida/content/Splunk-Standard-Security/modelregistry/offlineworkflow/ 

   5. Restart all services:

      /opt/caspida/bin/Caspida stop-all 
      /opt/caspida/bin/Caspida start-all

6. After setup completes:
   1. Open a web browser and log in to the public IP address with the default admin credentials to confirm a successful installation. The default username is admin and password is changeme. See Secure the default account after installing Splunk UBA for information about the default accounts provided with Splunk UBA and how to secure them.
   2. See Verify successful installation for more information about verifying a successful installation.