Send Splunk UBA anomalies and threats to Splunk ES as notable events

You can send anomalies and threats identified in Splunk User Behavior Analytics (UBA) to Splunk Enterprise Security (ES) as notable events.

The following image summarizes the steps you must take to send anomalies and threats from Splunk UBA to Splunk ES as notable events.

The Splunk Add-on for Splunk UBA is installed by default with Splunk ES and consists of two separate add-ons: SA-UEBA and the Splunk Add-on for UEBA.

Complete the following steps:

1. Verify that the Splunk Add-on for UEBA is installed and enabled.
2. Enable SA-UEBA so that Splunk ES can retrieve data from Splunk UBA.
3. Configure the Splunk platform to receive data from the Splunk UBA output connector.
4. Add an output connector in Splunk UBA.
5. (Optional) Learn How threats and notables are synchronized.

Verify that the Splunk Add-on for UEBA is installed and enabled

Perform the following steps to verify that the Splunk Add-on for UEBA is installed.

1. Log in to the Splunk search head with Splunk Enterprise Security installed.
2. In Splunk Web, select Apps > Manage Apps.
3. Search for ueba and verify that Splunk_TA_ueba is installed and enabled.
4. If the add-on is not enabled, click Enable to enable it.

Enable SA-UEBA so that Splunk ES can retrieve data from Splunk UBA

Enable SA-UEBA so that dashboards and knowledge objects are visible to users in Splunk Enterprise Security. This task is required to retrieve data from Splunk UBA to display on the Session Center dashboard in Splunk Enterprise Security, to model data sent from Splunk UBA with the UEBA data model, and to use correlation searches to analyze threats and anomalies sent from Splunk UBA.

1. Log in to the Splunk search head with Splunk Enterprise Security installed.
2. In Splunk Web, select Apps > Manage Apps.
3. Locate the SA-UEBA add-on.
4. Click Enable to enable the add-on.

Configure the Splunk platform to receive data from the Splunk UBA output connector

The connection between Splunk UBA and the Splunk platform uses TCP-SSL by default. Set up the Splunk platform to accept the encrypted connection so that the Splunk platform can receive data from the Splunk UBA output connector.

Splunk Cloud Platform customers must work with Splunk Cloud Platform Support to set up this connection.

The following procedure uses the Splunk default certificates and the global [SSL] stanza in the inputs.conf file. For better security, consider using your own certificates, or commercially signed certificates from a trusted certificate authority.

• See About securing Splunk Enterprise with SSL in the Splunk Enterprise Securing the Splunk Platform manual.
• See TCP: in the Splunk Enterprise Admin Manual for more information about configuring tcp-ssl using inputs.conf.

Steps on the Splunk Enterprise search head

Perform the following steps on the Splunk Enterprise search head. In a search head clustering environment, perform the changes on the search head that will receive UBA threats and anomalies:

1. Create a local folder under $SPLUNK_HOME/etc/apps/Splunk_TA_ueba. For example:

   cd /opt/splunk/etc/apps/Splunk_TA_ueba
   mkdir local
   cd local
   

2. Create a file called inputs.conf and add the following stanza:

   [tcp-ssl:10008]
   listenOnIPv6 = no
   index = ueba
   sourcetype = ueba
   serverCert = $SPLUNK_HOME/etc/auth/server.pem
   sslPassword = password
   




3. Restart Splunk Enterprise.
   1. In Splunk Web, select System > Server controls.
   2. Click Restart Splunk.
4. Verify that SSL is enabled for port 10008 in $SPLUNK_HOME/var/log/splunk/splunkd.log, as shown in the following examples:

   11-07-2019 15:07:42.661 -0800 INFO  TcpInputProc - Creating raw Acceptor for IPv4 port 10008 with SSL

   OR

   02-28-2024 22:34:37.330 +0000 INFO  TcpInputConfig [3996510 TcpListener] - IPv4 port 10008 is reserved for raw input (SSL)

5. Copy the root CA certificate from /opt/splunk/etc/auth/cacert.pem on the Splunk Enterprise instance to /home/caspida on the Splunk UBA management server.

   If you use your own third-party certificate, copy that certificate file to /home/caspida on the Splunk UBA management server. Do not copy the root CA certificate from /opt/splunk/etc/auth/cacert.pem on the Splunk Enterprise instance.

Steps on the Splunk UBA management server

Perform the following steps on the Splunk UBA management server:

1. Log in to the Splunk UBA management server as the caspida user.
2. Ensure that $JAVA_HOME is set correctly on your system. Run the CaspidaCommonEnv.sh script to set this environment variable:

   . /opt/caspida/bin/CaspidaCommonEnv.sh

3. Import the rootCA certificate to the Java certificate store.
   

   If you use your own third party certificate, replace ~/cacert.pem with that third party certificate in the following commands.

   On RHEL systems, use the following command:

   sudo keytool -import -alias "splunk es" -keystore $JAVA_HOME/lib/security/cacerts -file ~/cacert.pem

   On other Linux systems, use the following command:

   sudo keytool -import -alias "splunk es" -keystore $JAVA_HOME/jre/lib/security/cacerts -file ~/cacert.pem

4. When prompted, type the keystore password and trust the certificate. The default keystore password is changeit.
5. From the command line of the Splunk UBA management server, view the /etc/caspida/local/conf/uba-site.properties file to confirm the following parameter is set to "true" as shown:
   connectors.output.splunkes.ssl=true
6. Restart Splunk UBA. Run the following commands on the Splunk UBA management server:

   /opt/caspida/bin/Caspida stop-all
   /opt/caspida/bin/Caspida start-all
   

Add an output connector in Splunk UBA

Set up Splunk UBA to send data to the Splunk search head or a Splunk forwarder. Splunk UBA uses the output connector to send anomalies and threats to Splunk Enterprise Security.

1. Make sure the Splunk Add-on for Splunk UBA is properly deployed and enabled. See Deploy the Splunk Add-on for Splunk UBA.
2. Make sure the SA-UEBA add-on is enabled. See Enable SA-UEBA so that Splunk ES can retrieve data from Splunk UBA.
3. Log in to Splunk UBA as an admin user.
4. Select Manage > Output Connectors.
5. Click New Output Connector.
6. Select an output connector of SplunkES and click Next.
7. Type a Name for the output connector.
8. The Forwarder Host is the IP address of the Splunk search head or forwarder with the TCP data input enabled. When sending events to a Splunk ES search head cluster, choose one of the members to specify when configuring the Output Connector in Splunk UBA.

   Splunk UBA does not support connecting to a Splunk ES search head cluster through a load balancer.

9. The Forwarder Port is TCP port 10008 that is open for receiving data on the Splunk search head or forwarder. Make sure this port is open behind your firewall.
10. Specify the Splunk ES API URL in the following format:

    https://<forwarder-host>:<mgmt-port>

    Splunk UBA uses this connection to synchronize the status of its threats with notable events in Splunk ES. The management port is usually 8089.
11. Specify a username and password. See Splunk Enterprise and Splunk ES requirements for the requirements of this user account.
12. Select how you want this output connector to process anomalies and threats.

    Option	Description
    Process Threats	Select Process Threats to enable Splunk UBA to export threats based on any configured output connectors, such as sending email or sending threats to Splunk ES. If this is not selected, no threats are exported from Splunk UBA, even if Auto Process is selected.
    Process Anomalies	Select Process Anomalies to enable Splunk UBA to export anomalies based on any configured output connectors, such as sending email or sending anomalies to Splunk ES. If this is not selected, no anomalies are exported from Splunk UBA, even if Auto Process is selected.
    Auto Process	Select Auto Process to automatically forward anomalies and threats as they are created to Splunk ES. If this is not selected, you must forward each threat and corresponding anomalies during threat review with the Export to Splunk ES action from the Threat Details page. See Review Current Threats in Use Splunk User Behavior Analytics. Selecting Auto Process in conjunction with Process Anomalies can cause a significant increase in traffic. Review the output connector logs to verify the health and performance of your system. See the Output Connector Service health monitor status codes. If you are monitoring Splunk UBA using the Splunk UBA Monitoring app, see About the Splunk UBA Monitoring app in theSplunk UBA Monitoring App manual.

13. Click OK to save the new output connector.

If the output connector does not work, verify that you configured the host server correctly in the uba-site.properties file. The host identified in the file populates the host field for events sent to Splunk Enterprise Security or the Splunk platform. See Connect Splunk UBA to Splunk Enterprise to view an anomaly's raw events in Install and Upgrade Splunk User Behavior Analytics.

How threats and notable events are synchronized

Once the output connector is configured, Splunk UBA attempts to send threats to Splunk ES every 5 minutes with no limits on the number of retries. Any issues with the connection mean that new threats are not sent to Splunk ES until the connection issues are resolved. Any connection issues between the output connector and Splunk ES also affect other output connectors that are configured, such as email and ServiceNow. If the connection issues persist for more than one hour, alerts are generated in the health monitor in Splunk UBA. See Monitor the health of your Splunk UBA deployment in Administer Splunk User Behavior Analytics.

Perform the following steps to change the retry interval:

1. On the Splunk UBA management node, log in as the caspida user.
2. Edit the /etc/caspida/local/conf/uba-site.properties file and modify the uba.splunkes.retry.delay.minutes property. For example, to set the retry interval to 3 minutes:

   uba.splunkes.retry.delay.minutes = 3

3. In distributed deployments, synchronize the cluster:

   /opt/caspida/bin/Caspida sync-cluster  /etc/caspida/local/conf

Work with Splunk UBA threats as notable events in Splunk ES

When Splunk UBA and Splunk ES are integrated using an output connector, Splunk UBA creates a new custom status on Splunk ES called Closed in Uba as shown in the following image.

The status of threats in Splunk UBA and their corresponding notable events in Splunk ES are synchronized.

What happens when a threat is closed in Splunk UBA

Threats in Splunk UBA can be closed by the user or closed by the system:

• Threats in Splunk UBA are considered closed by the user if a user clicks Not a Threat in Splunk UBA.
• In all other cases, the threat in Splunk UBA is considered to be closed by the system.

When a threat is closed in Splunk UBA, Splunk UBA checks the status of the corresponding notable event in Splunk ES. If the notable event is not already closed in Splunk ES, Splunk UBA closes the notable event by setting the end status to Closed in Uba.

If the notable event is reopened in Splunk ES, a threat closed by the user in Splunk UBA is reopened. A threat closed by the system remains closed in Splunk UBA. The threat can still be viewed, but no action can be taken on the threat. This workflow is illustrated in the following diagram:

What happens when a threat is reopened in Splunk UBA

A threat in Splunk UBA can be reopened in the following cases:

• Threat computation causes a threat to be reopened.
• An anomaly action rule affects anomalies that cause a threat to be reopened.
• A threat rule is modified, causing a threat to be reopened.

When a threat is reopened, Splunk UBA checks to see if the notable event in Splunk ES has an end status of ClosedInUba and if yes, the notable event is also reopened.

No action is taken if the notable event is already open in Splunk ES, or if it has an end status other than ClosedInUba.

Splunk UBA queries for the status of notable events in Splunk ES

Splunk UBA queries Splunk ES in 5-minute intervals to synchronize the status of threats in Splunk UBA and notable events in Splunk ES.

When the query detects that a notable event is closed, Splunk UBA checks to see if the corresponding threat is also closed. If not, the threat is closed with a status of ClosedbyUser as shown in the following diagram:

When the query detects that a notable event is not closed, Splunk UBA checks to see if the corresponding threat has a status of ClosedbyUser. If so, the threat is reopened with a status of Active as shown in the following diagram:

What happens if the output connector is unable to send threats to Splunk ES?

If the output connector is unable to send threats to Splunk ES, due to a network issue or Splunk ES being temporarily unavailable, the output connector makes another attempt every 5 minutes. After 1 hour, if the connection is not resolved, the output connector raises an error in the health monitor. See OCS-11 in Administer Splunk User Behavior Analytics.