Install Splunk UBA on several Linux servers

Install Splunk UBA on several servers with Oracle Enterprise Linux (OEL) or Red Hat Enterprise Linux (RHEL) installed. Splunk UBA 5.4.0 requires OEL 8.8 or RHEL 8.6 or 8.8.

Follow these instructions to install Splunk UBA 5.4.0 for the first time. If you already have Splunk UBA, do not follow the instructions on this page. Instead, follow the appropriate upgrade instructions to obtain your desired release. See, How to install or upgrade to this release of Splunk UBA.

Prerequisites for installing Splunk UBA on several Linux servers

You must install Splunk UBA on a server that is running a supported operating system. See, Operating system requirements.
Make sure your Red Hat Enterprise Linux license includes the proper subscription names. See, Additional RHEL requirements.
Determine the interface of your system network configuration, for example eth0 or en0. You will need this information later in the installation process.

Configure permissions for and prepare the caspida user

Enable sudo permissions for the caspida user.

Use the visudo command to edit the /etc/sudoers file.
If the following line exists, comment the line Defaults requiretty.
Add the following lines at the end of the /etc/sudoers file.
```
caspida ALL=(ALL) NOPASSWD:ALL
Defaults secure_path = /sbin:/bin:/usr/sbin:/usr/bin
```
The /etc/sudoers file is read sequentially, so placing these lines at the end ensures that there is no impact to the caspida user from any existing accounts or group permissions.
Add the caspida user to the systems. The caspida user needs to have the same UID and GID on all the systems. Pick a UID and GID that is available on all the systems. For example, assuming UID and GID 2018 is available on all nodes:
```
groupadd --gid 2018 caspida
useradd --uid 2018 --gid 2018 -m -d /home/caspida -c "Caspida User" -s /bin/bash caspida
```
Set the password for caspida user:
```
passwd caspida
```
Verify the caspida user permissions for newly created files and directories. See Validate the UMASK value.

Obtain the installation package

Download the following Splunk UBA software and RHEL packages.

Obtain the Splunk UBA 5.4.0 software:
1. Go to the Splunk UBA Software Installation Package page on Splunkbase.
2. Download the file to the /home/caspida directory. The name of the package is splunk-uba-software-installation-package_540.tgz.

Use these packages for all supported Linux environments.

Prepare all servers in your distributed Linux environment

Perform these steps on every server node in the distributed deployment.

From the command line, log in to the server as the root user, or log in as a different user then use su or sudo to gain root user privileges.
Double-check your hostnames prior to running the setup. Verify the hostname of all the nodes using the following command:
```
hostname
```
Make sure all the nodes have a consistent setup. If using fully qualified domain names (FQDN) then all nodes should output FQDN in the hostname command. If the short name is used, then all nodes should output the short name in the hostname command.
If you plan on connecting to Splunk Cloud to run queries for datasources, use fully qualified domain names (FQDN), not short names, for your Splunk UBA hostnames.
Find the additional 1TB disk or disks using the fdisk -l command. The nodes that are running Spark services should have two 1TB disks. See Disk space and memory requirements for a summary of where Spark is running per deployment.
For example, you may see two disks named /dev/sdb and /dev/sdc.
Partition and format the partition on each disk found in step 2.
1. Partition and format the partition on the /dev/sdb disk using the following series of commands. Verify that the align-check opt 1 command returns 1 aligned.
```
parted -a optimal /dev/sdb
  mklabel gpt
  mkpart primary ext4 2048s 100%
  align-check opt 1
  quit
```
2. Format the partition using the mkfs command.
```
mkfs -t ext4 /dev/sdb1
```
3. On nodes that have a second disk, repeat the commands to partition and format the partition on /dev/sdc:
```
parted -a optimal /dev/sdc
  mklabel gpt
  mkpart primary ext4 2048s 100%
  align-check opt 1
  quit
```
4. Format the partition using the mkfs command. When prompted, confirm that you want to continue.
```
mkfs -t ext4 /dev/sdc1
```
Get the block ID for each disk using the blkid command. For example, to get the block IDs for /dev/sdb1 and /dev/sdc1 in our example:
```
blkid -o value -s UUID /dev/sdb1
blkid -o value -s UUID /dev/sdc1
```
An example block ID might be: 5c00b211-e751-4661-91c4-60d9f9315857.
Create new /var/vcap and /var/vcap2 directories. For example, on a node with a single 1TB disk:
```
mkdir -p /var/vcap
```
Or on a node with two 1TB disks:
```
mkdir -p /var/vcap /var/vcap2
```

Add the block ID for the /var/vcap partition to the /etc/fstab directory. For example, on a node with a single 1TB disk:

UUID=e1af8814-9b12-4c69-a947-18af370c7dd1 /var/vcap  ext4  defaults  0 0

On a node with two 1TB disks:

UUID=e1af8814-9b12-4c69-a947-18af370c7dd1 /var/vcap  ext4  defaults  0 0
UUID=f142f182-27c6-4002-b0bb-941fbedce17d /var/vcap2  ext4  defaults  0 0

Mount the file systems using the mount -a command.

Verify that the 1TB disks are mounted correctly using the df -h command. For example:

root# df -h
Filesystem      Size  Used Avail Use% Mounted on
...
/dev/sdc1       493G   77M  467G   1% /var/vcap2
/dev/sdb1       985G   43G  892G   5% /var/vcap
...

Inherit the permissions for the root user. On a node with a single 1TB disk:

chmod 755 /var/vcap
chown root:root /var/vcap

Or on a node with two 1TB disks:

chmod 755 /var/vcap /var/vcap2
chown root:root /var/vcap /var/vcap2

Make a directory for caspida software packages.
This should be different from caspida home directory (/home/caspida).
```
mkdir /opt/caspida
chown caspida:caspida /opt/caspida
chmod 755 /opt/caspida
```
Set the following environment variables for PostgreSQL in the /etc/locale.conf file:
```
LANG="en_US.UTF-8"
LC_CTYPE="en_US.UTF-8"
```
Run the following command to source the /etc/locale.conf file:
```
source /etc/locale.conf
```
Verify that the host name resolves using the nslookup <host name> command. If it does not, verify your host name lookup and DNS settings. See Configure host name lookups and DNS.
Verify that the system date, time and time zone are correct using the timedatectl command, as shown here. The time zone in Splunk UBA should match the time zone configured in Splunk Enterprise.
```
root# timedatectl status
      Local time: Mon 2019-04-08 14:30:02 UTC
  Universal time: Mon 2019-04-08 14:30:02 UTC
        RTC time: Mon 2019-04-08 14:30:01
       Time zone: UTC (UTC, +0000)
     NTP enabled: yes
NTP synchronized: yes
 RTC in local TZ: no
      DST active: n/a
```
Use the timedatectl command to change the time zone. For example, to change the time zone to UTC:
```
timedatectl set-timezone UTC
```
Refer to the documentation for your specific operating system to configure NTP synchronization. Use the ntpq -p command to verify that NTP is pointing to the desired server.
Modify /etc/sysconfig/selinux and set SELINUX=permissive.
With SELINUX set to enforced, certain actions during installation and upgrade (for example, access to particular files) can be blocked. Set SELINUX to permissive to allow Splunk UBA the necessary access so that actions are not blocked, but instead logged in the audit logs.

Verify that /proc/sys/net/bridge/bridge-nf-call-iptables exists on your system and the content of bridge-nf-call-iptables is 1. Run the following command to verify:

cat /proc/sys/net/bridge/bridge-nf-call-iptables

Your situation	Take this action
`/proc/sys/net/bridge/bridge-nf-call-iptables` exists on your system and the content is `1`.	Run the following command to make sure this setting is preserved through any reboot operations: echo net.bridge.bridge-nf-call-iptables=1 > /etc/sysctl.d/splunkuba-bridge.conf Go to Step 19.
`/proc/sys/net/bridge/bridge-nf-call-iptables` exists on your system but the content is not `1`.	Run the following commands to set the content of the `bridge-nf-call-iptables`: sysctl -w net.bridge.bridge-nf-call-iptables=1 Run the following command to ensure that the settings persist through any reboot operations: echo net.bridge.bridge-nf-call-iptables=1 > /etc/sysctl.d/splunkuba-bridge.conf Go to Step 19.
`/proc/sys/net/bridge/bridge-nf-call-iptables` does not exist on your system.	Run the following commands to create the file and ensure that it is loaded on reboot: modprobe br_netfilter echo br_netfilter > /etc/modules-load.d/br_netfilter.conf Run the following commands to create and set the content of the `bridge-nf-call-iptables`: sysctl -w net.bridge.bridge-nf-call-iptables=1 Run the following command to ensure that the settings persist through any reboot operations: echo net.bridge.bridge-nf-call-iptables=1 > /etc/sysctl.d/splunkuba-bridge.conf Go to Step 19.

Run the following command to ensure that /etc/sysctl.d/splunkuba-bridge.conf is readable by the caspida user:
```
chmod o+r /etc/sysctl.d/splunkuba-bridge.conf 
```
If that command returns without error, proceed to the next step.

Verify that IPv6 drivers are available. To do this, check that /proc/sys/net/ipv6/ exists. For example:

root# ls -l /proc/sys/net/ipv6/
total 0
-rw-r--r-- 1 root root 0 Mar 12 16:52 anycast_src_echo_reply
-rw-r--r-- 1 root root 0 Mar 12 16:52 auto_flowlabels
-rw-r--r-- 1 root root 0 Mar 12 16:52 bindv6only
dr-xr-xr-x 1 root root 0 Mar 12 16:52 conf
-rw-r--r-- 1 root root 0 Mar 12 16:52 flowlabel_consistency
-rw-r--r-- 1 root root 0 Mar 12 16:52 flowlabel_state_ranges
-rw-r--r-- 1 root root 0 Mar 12 16:52 fwmark_reflect
dr-xr-xr-x 1 root root 0 Mar 12 16:52 icmp
-rw-r--r-- 1 root root 0 Mar 12 16:52 idgen_delay
-rw-r--r-- 1 root root 0 Mar 12 16:52 idgen_retries
-rw-r--r-- 1 root root 0 Mar 12 16:52 ip6frag_high_thresh
-rw-r--r-- 1 root root 0 Mar 12 16:52 ip6frag_low_thresh
-rw-r--r-- 1 root root 0 Mar 12 16:52 ip6frag_secret_interval
-rw-r--r-- 1 root root 0 Mar 12 16:52 ip6frag_time
-rw-r--r-- 1 root root 0 Mar 12 16:52 ip_nonlocal_bind
-rw-r--r-- 1 root root 0 Mar 12 16:52 mld_max_msf
-rw-r--r-- 1 root root 0 Mar 12 16:52 mld_qrv
dr-xr-xr-x 1 root root 0 Mar 12 16:52 neigh
dr-xr-xr-x 1 root root 0 Mar 12 16:52 route
-rw-r--r-- 1 root root 0 Mar 12 16:52 xfrm6_gc_thresh

If the IPv6 drivers exist, skip to the next step.

If IPv6 drivers do not exist on your system, check if /etc/default/grub contains ipv6.disable=1. IPv6 drivers will not be available on a system if ipv6.disable=1 exists in /etc/default/grub. If ipv6.disable=1 is not present in /etc/default/grub and IPv6 drivers do not exist, consult with your system or network administrators. You will not be able to continue with the installation.

If /etc/default/grub contains ipv6.disable=1, perform the following tasks as root:

Remove ipv6.disable=1 from /etc/default/grub.

Recreate the grub config:

find /boot -name grub.cfg -exec grub2-mkconfig -o '{}' \;

Reboot the machines. After the system comes up, make sure /proc/sys/net/ipv6 exists.

To disable IPv6 functionality for security, networking or performance reasons, create the /etc/sysctl.d/splunkuba-ipv6.conf file as root. This file should contain the following content:

net.ipv6.conf.all.disable_ipv6 = 1
net.ipv6.conf.default.disable_ipv6 = 1
net.ipv6.conf.lo.disable_ipv6 = 1

This procedure keeps the IPv6 drivers but turns off IPv6 addressing.

Create the /etc/security/limits.d/caspida.conf file and add the following security limits for the caspida user to this file:

caspida soft nproc unlimited
caspida soft nofile 32768
caspida hard nofile 32768
caspida soft core unlimited
caspida soft stack unlimited
caspida soft memlock unlimited
caspida hard memlock unlimited

Make sure the root account does not have any security limits.

If you are not using IPv6 on your network, edit the /etc/yum.conf file and add the following entry so that only IPv4 addresses are used by yum/rpm:
```
ip_resolve=4
```
If you have any firewall configuration enabled, such as firewalld, disable the configuration during installation. Run the following command:
```
systemctl disable firewalld
```
You can re-enable your firewall settings after the setup is complete.
Restart the system.
```
init 6
```
After the system restarts, use the following command to verify that the host name matches your host name lookup and DNS settings. See Configure host name lookups and DNS.
```
hostname --fqdn
```

Turn on FIPS compliance

Turning on FIPS compliance is an optional step when installing or upgrading Splunk UBA.

Federal Information Processing Standard (FIPS) compliance is available with Splunk UBA version 5.4.0 and higher. Complete the following steps to turn on FIPS on each Splunk UBA node before running the install.sh script in the Install Splunk UBA on each Linux server section.

Run the following command to check the current status of FIPS:
```
sudo fips-mode-setup --check
```
On each node, run the following command to turn on FIPS:
```
sudo fips-mode-setup --enable
```
After successfully turning on FIPS, reboot the system:
```
sudo reboot
```
Confirm FIPS is turned on:
```
sudo fips-mode-setup --check
```
You can also verify the status using the following command.
You see a 1 if FIPS is turned on, otherwise 0.
```
cat /proc/sys/crypto/fips_enabled
```

Install Splunk UBA on each Linux server

Perform these steps on every server node in the distributed deployment to install Splunk UBA. If you are running the commands from the /home/caspida directory, you can omit the /home/caspida portion of the commands.

Log in to the command line as the caspida user using SSH.
Verify that the caspida user has umask permissions set to 0022 or 0002.
umask
If the returned values are not supported, edit the ~/.bash_profile and the ~/.bashrc files and append umask 0022
Verify that the splunk-uba-software-installation-package_540.tgz file that you downloaded in Obtain the installation package is in the /home/caspida directory. If not, copy the files to the /home/caspida directory.
Untar the file for Splunk UBA Software Installation in the /home/caspida directory.
```
tar xvzf /home/caspida/splunk-uba-software-installation-package_540.tgz
```
Run the following command to untar the Splunk UBA platform software to the /opt/caspida directory:
```
tar xvzf /home/caspida/Splunk-UBA-Platform-5.4.0-20240424-16474780.tgz -C /opt/caspida/
```
Run the following command to untar the Splunk UBA packages to the /home/caspida directory:
```
tar xvzf /home/caspida/Splunk-UBA-5.4-Packages-RHEL-8.tgz -C /home/caspida
```

Run the installation script.

/opt/caspida/bin/installer/redhat/INSTALL.sh /home/caspida/Splunk-UBA-5.4-Packages-RHEL-8

The log file is /var/log/caspida/install.log.

If you do not use One-Time-Password (OTP) or Multi-Factor Authentication (MFA) methods, and you see the following error, run the command sudo yum remove krb5-workstation.
If you do use OTP or MFA skip to sub-step b.

error: Failed dependencies:
krb5-libs(x86-64) = 1.18.2-22.el8_7 is needed by (installed) krb5-workstation-1.18.2-22.el8_7.x86_64
libkadm5(x86-64) = 1.18.2-22.el8_7 is needed by (installed) krb5-workstation-1.18.2-22.el8_7.x86_64

If you use One-Time-Password (OTP) or Multi-Factor Authentication (MFA) methods, and you see the following error, run the command sudo yum downgrade krb5-workstation.

error: Failed dependencies:
krb5-libs(x86-64) = 1.18.2-22.el8_7 is needed by (installed) krb5-workstation-1.18.2-22.el8_7.x86_64
libkadm5(x86-64) = 1.18.2-22.el8_7 is needed by (installed) krb5-workstation-1.18.2-22.el8_7.x86_64

Then re-run the installation script

Generate SSH keys using the ssh-keygen -t rsa command. Press enter for all the prompts and accept all default values. For example:

[caspida@ubahost-001]$ ssh-keygen -t rsa
Generating public/private rsa key pair.
Enter file in which to save the key (/home/caspida/.ssh/id_rsa):
Created directory '/home/caspida/.ssh'.
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in /home/caspida/.ssh/id_rsa.
Your public key has been saved in /home/caspida/.ssh/id_rsa.pub.
The key fingerprint is:
SHA256:Ohe1oSpUtNT8siJzvn2lFLrHmVH7JGKke+c/5NRFb/g caspida@ubahost-001

Add the SSH keys to the server and adjust the permissions to allow the server to access them.

cat /home/caspida/.ssh/id_rsa.pub >> /home/caspida/.ssh/authorized_keys
chmod 600 /home/caspida/.ssh/authorized_keys

Copy the SSH keys from every server from /home/caspida/.ssh/id_rsa.pub into the /home/caspida/.ssh/authorized_keys file on every server in the distributed deployment. After this is complete, each authorized_keys file on each server should have all the SSH keys for every server in the deployment listed.
Test to verify that the SSH connections do not require a password. Connect to each server without a password with the host name using SSH to create trusted connections between the servers. After you confirm the connection does not require a password, use exit to terminate the SSH connection. You must complete this step before continuing with setup.
```
ssh <node1>; exit
ssh <node2>; exit
ssh <node3>; exit
```

When prompted, confirm that you want to continue. For example, the sample output will look similar to the following.

[caspida@ubahost-001]$ ssh uba1
The authenticity of host 'uba1 (172.31.35.204)' can't be established.
ECDSA key fingerprint is af:12:54:60:f5:36:c2:36:9d:56:b2:52:9f:cb:73:bc.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added 'uba1,172.31.35.204' (ECDSA) to the list of known hosts.

Complete the distributed Linux Splunk UBA installation on the management server

After completing the previous steps on all server nodes in the distributed deployment, perform the following steps on the management server. For example, uba-node01.

Check the system status with the uba_pre_check.sh shell script.
If you used FQDN instead of short-names to setup UBA, use FQDNs when running the uba_pre_check.sh script.

You must specify the host name of each Splunk UBA node in the command, separated by spaces. For example, run the following command in a 3-node deployment and be sure to replace <node1> <node2> <node3> with the actual host names of your Splunk UBA nodes.
```
/opt/caspida/bin/utils/uba_pre_check.sh <node1> <node2> <node3>
```
See Check system status before and after installation for more information about the script.
Run the setup script.
```
/opt/caspida/bin/Caspida setup
```
1. When prompted, accept the license agreement and confirm removal of existing metadata.
2. When prompted, type a comma-separated list of host names for your distributed installation. For example, specify the following in a 3-node deployment and be sure to replace <node1> <node2> <node3> with the actual host names of your Splunk UBA nodes:
```
<node1>,<node2>,<node3>
```
3. When prompted for a list of host names in the setup script, if the output of the host name command is FQDN, then provide a CSV list of FQDN host names, for example:
```
<NODE1_FQDN>,<NODE2_FQDN>,<NODE3_FQDN>
```
4. When prompted, confirm that you want to continue setting up Splunk UBA.
5. The log file is /var/log/caspida/caspida.out

Ensure the proper configuration of the Splunk forwarder. Commands must be executed on the management node.

Stop UBA's Splunk forwarder:
```
/opt/caspida/bin/Caspida stop-splunk
```

Run the following command to update TCP connector properties names in outputs.conf to the correct format:

sed -i \
 -e 's/sslpassword/sslPassword/g' \
 -e 's/autolbfrequency/autoLBFrequency/g' \
 -e 's/autolbvolume/autoLBVolume/g' \
 -e 's/forcetimebasedautolb/forceTimebasedAutoLB/g' \
 -e 's/clientcert/clientCert/g' \
 -e 's/sslrootcapath/sslRootCAPath/g' \
 -e 's/sslcommonnametocheck/sslCommonNameToCheck/g' \
 -e 's/sslverifyservercert/sslVerifyServerCert/g' \
 -e 's/useclientsslcompression/useClientSSLCompression/g' \
 /opt/splunk/etc/apps/Splunk_UBA_Monitor/default/outputs.conf

Clean fishbucket to force re-indexing of the telemetry.data.out events file:

[ -f /var/log/caspida/telemetry/events/telemetry.data.out ] && /opt/splunk/bin/splunk cmd btprobe -d /opt/splunk/var/lib/splunk/fishbucket/splunk_private_db --file /var/log/caspida/telemetry/events/telemetry.data.out --reset

Start UBA's Splunk forwarder:
```
/opt/caspida/bin/Caspida start-splunk
```

Perform one final sync across your cluster:
```
/opt/caspida/bin/Caspida sync-cluster
```

Restart UBA:

/opt/caspida/bin/Caspida stop
/opt/caspida/bin/Caspida start

After setup completes:
1. Open a web browser and log in to the public IP address with the default admin credentials to confirm a successful installation. The default username is admin and password is changeme. See Secure the default account after installing Splunk UBA for information about the default accounts provided with Splunk UBA and how to secure them.
2. See Verify successful installation for more information about verifying a successful installation.

Related answers from Splunk Community

Install Splunk UBA on several Linux servers

Prerequisites for installing Splunk UBA on several Linux servers

Configure permissions for and prepare the caspida user

Obtain the installation package

Prepare all servers in your distributed Linux environment

Turn on FIPS compliance

Install Splunk UBA on each Linux server

Complete the distributed Linux Splunk UBA installation on the management server

Comments

Install Splunk UBA on several Linux servers

Was this topic useful?