Manage the number of threats and anomalies in your environment

The Offline Rule Executor in Splunk UBA runs nightly to process the scheduled anomaly and threat rules, and also performs threat revalidation in real time when there are rule changes, anomalies are removed from the system, or anomaly scores are changed. Threat revalidation can take a long time and cause memory issues on your system depending on a variety of factors, including the types and age of the anomalies involved in the threat, the number or anomalies and entities involved in the threat, and any custom threat rules active in the system.

Perform regular maintenance of your Splunk UBA deployment by managing the number of threats and anomalies in your system.

When deleting large number of anomalies, do not delete more than 200,000 anomalies at a time.

• Perform regular cleanup of anomalies more than 90 days old. See Delete anomalies in Splunk UBA.
• Close unwanted threats. See Close threats in Splunk UBA.
• Monitor the total number of anomalies in your environment.
  • If your deployment is fewer than 10 nodes, do not exceed 800,000 anomalies.
  • If your deployment is 10 nodes or more, do not exceed 1.5 million anomalies.
• Monitor the number of rule-based threats in your environment.
  • If your deployment is fewer than 10 nodes, do not exceed 1,000 rule-based threats.
• If your deployment is 10 nodes or more, do not exceed 2,000 rule-based threats.

The Offline Rule Executor times out in 15 minutes, meaning that if a threat rule takes longer than 15 minutes to complete, or threat revalidation takes longer than 15 minutes, some computations are lost and not generated in Splunk UBA. If a threat rule is taking longer than 15 minutes to complete, you can edit the rule parameters to try to shorten the time. See Monitor policy violations with custom threats.