Use Event Drilldown to review an anomaly's raw events

You can use the Event Drilldown panel on an Anomaly Details page to review raw events in Splunk Enterprise that led to anomaly creation in Splunk UBA.

Event Drilldown is only available for anomalies generated by models, not for rule-based anomalies.

Event Drilldown creates machine-generated SPL which serves as a starting point for your investigation into events that contributed to the anomaly. In this search, Splunk UBA identifies the data sources, users, devices, and associated metadata to form the SPL. Use this SPL as a starting point for your searches. The initial machine-generated SPL helps you to avoid having to write the SPL from scratch. It is expected that you will adjust the search based on contextual information in the anomalies.

Prerequisites for using Event Drilldown

Verify the following settings before using Event Drilldown.

Splunk UBA must be connected to Splunk Enterprise. To define the search heads and search head clusters in your Splunk Enterprise deployment, follow the instructions in Connect Splunk UBA to Splunk Enterprise to view an anomaly's raw events.

Any Splunk UBA user with any role can review an anomaly's raw events in Splunk Enterprise using their own credentials. The person viewing the events does not have to be the same person who on-boarded the data into Splunk Enterprise.

In some cases the generated URL may be too long to be processed by the Splunk HTTP server. If this is the case perform the following tasks:

1. Set the following property in the /etc/caspida/local/conf/uba-site.properties file in Splunk UBA:

   triggering.event.search.backend.submission = true

2. In distributed deployments, synchronize the cluster:

   /opt/caspida/bin/Caspida sync-cluster  /etc/caspida/local/conf

3. Restart the job manager:

   sudo service  caspida-jobmanager stop
   sudo service  caspida-jobmanager start
   

Completing the previous steps causes Splunk UBA to submit the generated SPL to Splunk Enterprise using the role of the data source. The search job permissions allow read access for any user belonging to the Splunk roles starting with uba_. For more information, see Manage user accounts and account roles in Splunk UBA.

Where to see Event Drilldown

The Event Drilldown panel appears on the Anomaly Details page. The following example shows the Event Drilldown panel for the Download from Internal Server anomaly, which is generated by an offline model:

The Event Drilldown panel for anomalies generated by streaming models such as Suspicious Network Connection are populated with a sample event, as shown in the example below:

Use and configure Event Drilldown

You can use Event Drilldown by performing the following tasks:

1. Click Generate Contributing Events Link(s) to generate a link called View Events Link. Splunk UBA pre-calculates this link for certain types of anomalies so that View Events Link is visible when the Anomaly Details page is loaded for the first time. See Splunk UBA caches SPL for important anomalies.
2. Click View Events Link to view raw events in Splunk Enterprise and begin your investigation. The Generate Contributing Events Link(s) button is inactive and will remain this way unless you change the Advanced Identity Lookup toggle. See Use Advanced Identity Lookup.

When there are events coming from search heads in multiple search head clusters to generate an anomaly, you see multiple View Events Link links after you click Generate Contributing Events Link(s).

When you click on View Events Link, an SPL search is automatically generated using relevant data from the anomaly, including username, email ID, internal and external IP addresses, available host names, and time range. Use this search to collect more supporting evidence from Splunk Enterprise. In environments with search heads in multiple clusters, each View Events Link link generates its own SPL search.

The first line of the generated SPL is used for event drilldown and cannot contain any macros or filters to subsearches such as [| inputlookups ...].

Make sure that the configured datasource SPL doesn't have the first PIPE (sub search query) inside the square parenthesis.

Perform the following steps if you need to edit your SPL:

1. Always wrap the square parenthesis with "( and )" wherever it's possible.
   Example:

   (index=*default sourcetype=newdatasource) NOT ([| inputlookup logging1.csv]) NOT ([| inputlookup logging2.csv]) NOT ([| inputlookup logging3.csv | rename dest as src]) | eval action="allowed", eventtype=category | fields action,alarmCategories,bytes,bytes_in,bytes_out,category, dest_host,dest_ip,dest_port,duration,eventtype, ids_type,severity,signature,sourcetype,src_host,src_ip,src_port,tag,user

2. Log in to the Splunk UBA management node.
3. Restart the job manager.

   sudo service caspida-jobmanager stop
   sudo service caspida-jobmanager start
   

Splunk UBA caches SPL for important anomalies

Splunk UBA caches the SPL for important anomalies. Raw events for the following types of anomalies are pre-populated and cached in Splunk UBA. When you load the Anomaly Details page, the View Events Link link is already visible for anomalies in either of the following categories:

• Anomalies included in a threat
• Anomalies with a score of 8 or higher

You can adjust the anomaly score threshold by performing the following tasks:

1. Configure the triggering.event.pre.calculate.links.anomaly.threshold property in the /etc/caspida/local/conf/uba-site.properties file to adjust the anomaly score threshold. The default is 8. As an example, you can set the threshold to 9 if you only want anomalies with a score of 9 or higher to be pre-populated and cached in Splunk UBA.
2. In distributed deployments, synchronize the cluster:

   /opt/caspida/bin/Caspida sync-cluster  /etc/caspida/local/conf

3. Restart the job manager:

   sudo service  caspida-jobmanager stop
   sudo service  caspida-jobmanager start
   

Use Advanced Identity Lookup

Enable the Advanced Identity Lookup toggle to view the original IDs in the raw events instead of the Splunk UBA entity name. For example, instead of seeing the resolved device name of jsmith-laptop, you want to see the actual IP address of the device instead to aid in your investigation.

If you have already generated a View Events Link by clicking Generate Contributing Events Link(s) and the Generate Contributing Events Link(s) is inactive, changing the Advanced Identity Lookup toggle makes the Generate Contributing Events Link(s) active again. You can click on Generate Contributing Events Link(s) to generate new links.

• Splunk UBA requires up to two hours to generate advanced identity data for events. If an anomaly is not yet two hours old, this toggle is disabled until the anomaly is more than two hours old.
• When the toggle is enabled, identity data from the past 7 days is used to view contributing events. If an anomaly is more than 7 days old and event drilldown links have not been calculated before, this toggle is disabled and cannot be changed.
• For anomalies that already have calculated links, the toggle appears in the position that was used to generate the links. You can change the toggle position and generate new links.

Configure properties to increase the timeout interval

If you see a "Triggering Event time out" error message, perform the following tasks:

1. Adjust the time out value in the triggering.event.timeout.millis property in the /etc/caspida/local/conf/uba-site.properties file. The default timeout value is 5 minutes, or 300,000 milliseconds.
2. In distributed deployments, synchronize the cluster:

   /opt/caspida/bin/Caspida sync-cluster  /etc/caspida/local/conf

3. Restart the job manager:

   sudo service  caspida-jobmanager stop
   sudo service  caspida-jobmanager start
   

Configure reverse IR to view contributing events

Identity resolution (IR) allows Splunk UBA to perform real-time identity resolution to accurately define users and devices. Reverse IR enables you to view the contributing events for the IP address of a device even if the device name is resolved through IR. However, reverse IR generates additional overhead on the system and can affect performance when viewing contributing events, potentially leading to timeouts.

Reverse IR is enabled by default. To disable reverse IR, perform the following steps:

1. Set the triggering.event.enable.reverse.ir property in the /etc/caspida/local/conf/uba-site.properties file to false:

   triggering.event.enable.reverse.ir = false

2. In distributed deployments, synchronize the cluster:

   /opt/caspida/bin/Caspida sync-cluster  /etc/caspida/local/conf

3. Restart the job manager:

   sudo service  caspida-jobmanager stop
   sudo service  caspida-jobmanager start
   

Configure additional search term strategy

When generating contributing event links of particular anomaly, Splunk UBA appends additional search terms of the associated anomaly in the final SPL query.

Splunk UBA offers two search strategy options, Broad and Strict. By default, Splunk UBA uses Broad strategy between additional search terms in the SPL query.

To decide which strategy is used for additional search terms perform the following steps:

1. Set the triggering.event.additional.search.terms.strategy property in the /etc/caspida/local/conf/uba-site.properties file.
   • Use Broad value for this property to reflect OR as a conjunction between search terms in the SPL query. For example: (Search query) AND (AdditionalSearchTerm1 OR AdditionalSearchTerm2)

     triggering.event.additional.search.terms.strategy = Broad

   • Use Strict value for this property to reflect AND as a conjunction between search terms in the SPL query. For example: (Search query) AND (AdditionalSearchTerm1 AND AdditionalSearchTerm2)

     triggering.event.additional.search.terms.strategy = Strict

2. In distributed deployments, synchronize the cluster:

   /opt/caspida/bin/Caspida sync-cluster  /etc/caspida/local/conf

3. Restart the job manager:

   sudo service caspida-jobmanager stop

   sudo service caspida-jobmanager start

Configure additional search term appending strategy

When generating contributing event links of particular anomaly, Splunk UBA appends additional search terms of associated anomaly in the final SPL query.

Splunk UBA offers two search strategy options, Broad and Strict. By default, Splunk UBA uses Broad strategy for appending additional search terms to the main SPL query.

To decide which strategy is used for appending additional search terms to the main query perform following steps:

1. Set the triggering.event.additional.search.terms.spl.strategy property in the /etc/caspida/local/conf/uba-site.properties file.
   • Use Broad value for this property to reflect OR as a conjunction between search query and additional search terms in the SPL query. For example: (Search query) OR (AdditionalSearchTerm1 OR AdditionalSearchTerm2)

     triggering.event.additional.search.terms.spl.strategy = Broad

   • Use Strict value for this property to reflect AND as a conjunction between search query and additional search terms in the SPL query. For example: (Search query) AND (AdditionalSearchTerm1 OR AdditionalSearchTerm2)

     triggering.event.additional.search.terms.spl.strategy = Strict

2. In distributed deployments, synchronize the cluster:

   /opt/caspida/bin/Caspida sync-cluster  /etc/caspida/local/conf

3. Restart the job manager:

   sudo service  caspida-jobmanager stop

   sudo service  caspida-jobmanager start