Manage the number of threats and anomalies in your environment

The Offline Rule Executor in Splunk UBA runs nightly to process the scheduled anomaly and threat rules, and also performs threat revalidation in real time when there are rule changes, anomalies are removed from the system, or anomaly scores are changed. Threat revalidation can take a long time and cause memory issues on your system depending on a variety of factors, including the types and age of the anomalies involved in the threat, the number or anomalies and entities involved in the threat, and any custom threat rules active in the system.

Perform regular maintenance of your Splunk UBA deployment by performing any combination of the following tasks as needed:

• Perform regular cleanup of anomalies more than 90 days old. See Delete anomalies in Splunk UBA.
• Close unwanted threats. See Close threats in Splunk UBA.
• Monitor the total number of anomalies in your environment.
  • If your deployment is fewer than 10 nodes, do not exceed 1 million anomalies. For Splunk UBA releases 5.0.3 and earlier, do not exceed 800,000 anomalies.
  • If your deployment is 10 nodes or more, do not exceed 1.5 million anomalies.
• Monitor the number of rule-based threats in your environment.
  • If your deployment is fewer than 10 nodes, do not exceed 1,000 rule-based threats.
  • If your deployment is 10 nodes or more, do not exceed 2,000 rule-based threats.
• If you have some threat rules which require more than one hour to run, modify the rule engine timeout period.
  1. Log in to the Splunk UBA management node as the caspida user.
  2. Add or edit the rule.engine.process.timeout.min property to /etc/caspida/local/conf/uba-site.properties and the the desired number of minutes. The default is 60 minutes. The following example sets the timeout period to 90 minutes:

     rule.engine.process.timeout.min=90

  3. In distributed Splunk UBA deployments, run the following command on the management node to synchronize the cluster:

     /opt/caspida/bin/Caspida sync-cluster /etc/caspida/local/conf

  4. Run the following command on the management node to restart the Offline Rule Executor:

     sudo service caspida-offlineruleexec restart