Troubleshoot your Splunk UBA installation

This section contains information to help you analyze activity and diagnose problems with your Splunk UBA deployment.

Chrome browser reports ERR_CERT_COMMON_NAME_INVALID

In Splunk UBA 4.1.x and 4.2.x releases, you see the "ERR_CERT_COMMON_NAME_INVALID" message in the Chrome browser.

Cause

The Subject Alternate Names are missing, and the OpenSSL configuration does not ask for Subject Alternate Names.

Solution

Generate the CSR with the Subject Alternate Names, then the CSR must be signed by the Root CA.

1. Create the /opt/caspida/conf/deployment/templates/local_conf/ssl directory:

   mkdir /opt/caspida/conf/deployment/templates/local_conf/ssl

2. Create a file named openssl-altname.cnf in this directory. The file must contain the following:

   #######
   [ req ]
   default_bits       = 2048
   distinguished_name = req_distinguished_name
   attributes        = req_attributes
   
   [ req_distinguished_name ]
   countryName                = Country Name (2 letter code)
   stateOrProvinceName        = State or Province Name (full name)
   localityName               = Locality Name (eg, city)
   organizationName           = Organization Name (eg, company)
   organizationalUnitName     = Organizational Unit Name (eg, section)
   commonName                 = Common Name (e.g. server FQDN or YOUR name)
   
   [req_attributes]
   subjectAltName             = Alternative DNS names, Email adresses or IPs (comma separated, e.g DNS:test.example.com, DNS:test, IP:192.168.0.1)
   #####
   

3. Generate the CSR:

   openssl req -sha256 -new -key my-server.key.pem -out myCACertificate.csr -config /opt/caspida/conf/deployment/templates/local_conf/ssl/openssl-altname.cnf

   When prompted for the alternative DNS names, enter them separated by commas. For example:

   DNS:test.example.com, DNS:test, IP:192.168.0.1

   The IP address is required if you will be using it to access Splunk UBA.
4. Verify that the CSR contains the Subject Alternate Names:

   openssl req -text -noout -verify -in myCACertificate.csr | grep DNS

5. Sign this CSR with the root CA.

You see the "setup containerization failed" error

The setup-containerization command in /opt/caspida/bin/Caspida is run as part of the Splunk UBA setup or upgrade procedure. If the script fails, you will see the error message "setup containerization failed".

If you see this error:

Cause	Solution
Check the /var/log/caspida/caspida.out log file and fix the issues causing the command to fail. Then, run the setup-containerization command again. Repeat this process as needed until the command runs without producing any errors.	After setup-containerization runs without errors: If you are installing Splunk UBA for the first time, run: /opt/caspida/bin/CaspidaCleanup -i rules The CaspidaCleanup command removes all data from Splunk UBA. Run this command only if you are installing Splunk UBA for the first time, or if you are OK with removing all data from the system. If you are upgrading from an earlier release of Splunk UBA, run: /opt/caspida/bin/Caspida start-all If the Caspida start-all command fails to run, use the following command: /opt/caspida/bin/Caspida start-containers
The root disk may be more than 90% full.	Free up additional disk space and re-run the setup-containerization command.
HTTP or HTTPS proxy servers are configured, causing the setup-containerization command to fail.	To bypass HTTP/HTTPS proxy configuration, use any text editor to set the no_proxy and NO_PROXY environment variables in the /etc/environment file. See Modify firewalls and proxies. If the /etc/systemd/system/docker.service.d/http-proxy.conf file also exists on your system, perform the following tasks to update the http-proxy.conf file: Run the following commands: echo Environment=\"${HTTP_PROXY}\" echo Environment=\"${HTTPS_PROXY}\" echo Environment=\"${NO_PROXY}\" Copy the output of the above commands it to the /etc/systemd/system/docker.service.d/http-proxy.conf file. The backslashes are required so that quotation marks in the output are maintained. Below is an example of the output: [root@ubanode01]# echo Environment=\"${HTTP_PROXY}\" Environment="http://1.2.3.4:3128" [root@ubanode01]# echo Environment=\"${HTTPS_PROXY}\" Environment="http://1.2.3.4:3128" [root@ubanode01]# echo Environment=\"${NO_PROXY}\" Environment="localhost,127.0.0.1,127.0.1.1,127.0.0.0/8,10.96.0.0/12,10.244.0.0/16,172.17.0.1,172.17.0.2,172.17.0.0/16,ubanode01,ubanode01.mydomain.local,10.10.10.1,ubanode02,ubanode02.mydomain.local,10.10.10.2,ubanode03,ubanode03.mydomain.local,10.10.10.3,ubanode04,ubanode04.mydomain.local,10.10.10.4,ubanode05,ubanode05.mydomain.local,10.10.10.5,ubanode06,ubanode06.mydomain.local,10.10.10.6,ubanode07,ubanode07.mydomain.local,10.10.10.7,ubanode08,ubanode08.mydomain.local,10.10.10.8,ubanode09,ubanode09.mydomain.local,10.10.10.9,ubanode10,ubanode10.mydomain.local,10.10.10.10"

You see the "Warning: rebuild-uba-images failed" message

When installing Splunk UBA 4.2.x content updates, you may see the "Warning" rebuild-uba-images failed" message in Splunk UBA.

Cause

The content update installer is unable to rebuild the container images.

Solution

Perform the following steps if you see this message:

1. Stop the containers.

   /opt/caspida/bin/Caspida stop-containers

2. Rebuild the Splunk UBA images.

   /opt/caspida/bin/Caspida rebuild-uba-images

3. Start the containers.

   /opt/caspida/bin/Caspida start-containers

Changes to resolv.conf no longer stick

Changes to resolv.conf are not sticking.

Cause

The resolv.conf file is being overwritten by a network manager device.

Solution

On RHEL 7.x, perform the tasks in the following example, using eth0 as the interface and mydomain.com as the DNS suffix you want to add:

1. Check the search suffix setting.

   cat /etc/resolv.conf

2. Edit the file /etc/sysconfig/network-scripts/ifcfg-eth0 and append the following line:

   SEARCH="mydomain.com"

3. Restart the interface:

   sudo ifdown-eth0 && sudo ifup-eth0

4. Verify that the new DNS suffix has been added:

   cat /etc/resolv.conf

Rebalance HDFS storage to manage disks getting full in distributed deployments

You have disks in your multi-node Splunk UBA deployment that are filling up, such as exceeding 90% in capacity.

Cause

Splunk UBA uses HDFS for storing analytics data. Over time, the storage of this data across the data nodes in your Splunk UBA cluster can become imbalanced. Splunk UBA does not automatically rebalance HDFS storage across your cluster, meaning that some nodes may have much higher HDFS storage than others.

Solution

Perform the following tasks to examine how HDFS storage is distributed throughout your cluster.

1. Log in to the management node as the caspida user.
2. Run the /opt/caspida/bin/utils/uba_health_check.sh script. The output is saved to /var/log/caspida/check.
3. Edit the saved output and search for hdfs dfsadmin.
4. Examine for the DFS Used% percentage on each node. You can consider rebalancing HDFS if you have a large difference among the nodes. For example, one node is using 80% of its HDFS capacity, while another is using 55%.
5. To manually rebalance HDFS storage, use the following command to specify the maximum difference in HDFS storage percentage among all nodes. For example, to specify that HDFS storage should not differ by more than 5 percent on all nodes, use the following command:

   sudo -u hdfs hdfs balancer -threshold 5

HDFS rebalancing occurs on all the data nodes in your cluster. You can view your deployment configuration in the /opt/caspida/conf/deployment/recipes/deployment-<number_of_nodes>.conf file. For example, in a seven node cluster, look in /opt/caspida/conf/deployment/recipes/deployment-7_node.conf. The data nodes are defined in the hadoop.datanode.host property. See Where services run in Splunk UBA in Administer Splunk User Behavior Analytics.