Splunk® User Behavior Analytics

Administer Splunk User Behavior Analytics

Acrobat logo Download manual as PDF


Acrobat logo Download topic as PDF

How Splunk UBA synchronizes the primary and standby systems

Critical data stored in the Postgres database, such as the threats and anomalies generated by Splunk UBA rules and models, is synchronized in real-time.

All other data, such as the data stores in HDFS, some metadata in Postgres, along with the data in Redis and InfluxDB, is synchronized every four hours. A checkpoint is created for each sync. When there is a failover, the standby Splunk UBA system begins to replay data ingestion from the last available checkpoint. The Splunk platform might retain new events for Splunk UBA to consume, but the search performance might be different depending on the time between the last checkpoint and when the standby system begins ingesting events. For example, some events in the Splunk platform might have been moved to the cold bucket, which negatively affects the search performance. In addition, since the ingestion lag in Splunk UBA is configurable per data source, some raw events with time stamps beyond the configured lag are excluded from the replay.

Additional data loss can occur if, at the time of the failover, there are events in the data pipeline that are not yet consumed by Splunk UBA and therefore cannot persist in Splunk UBA. These events are lost and can't be recovered during a failover operation.

The initial sync of full data transfer is triggered automatically when next scheduled job starts, defined by ReplicationCoordinator in /etc/caspida/local/conf/caspida-jobs.json.

Resolve synchronization failures between the primary and standby systems

If the synchronization between the primary and standby systems fail, perform the following tasks to resume the synchronization operations:

  1. Run the following commands on the management node in the standby system:
    /opt/caspida/bin/CaspidaCleanup 
    /opt/caspida/bin/Caspida stop
    
  2. Run the following command on the management node in the primary system:
    /opt/caspida/bin/replication/setup standby -m primary -r
  3. Run the following command on the management node in the standby system:
    /opt/caspida/bin/replication/setup standby -m standby -r
  4. Wait until an entry in the replication table shows up on standby system. In all deployments smaller than 20 nodes, run the following command on the management node to check. In 20-node clusters, run the command on node 2:
    psql -d caspidadb -c 'select * from replication'
  5. Run the curl command on the management node in the primary system to initiate a full sync:
    curl -X POST -k -H "Authorization: Bearer $(grep '^\s*jobmanager.restServer.auth.user.token=' /opt/caspida/conf/uba-default.properties | cut -d'=' -f2)" https://localhost:9002/jobs/trigger?name=ReplicationCoordinator 
    
  6. Check the status of the replication table in both the primary and standby systems and verify the full cycle is triggered and there are no errors. Run the following command on the management node:
    tail -f /var/log/caspida/replication/replication.log
  7. Check the status of the replication table in both the primary and standby systems. After the replication is done, both the primary and standby systems have an Active status and the cycles should be synchronized. In all deployments smaller than 20 nodes, run the following command on the management node of both systems to check. In 20-node clusters, run the command on node 2 of both systems:
    psql -d caspidadb -c 'select * from replication'
Last modified on 09 November, 2021
PREVIOUS
Set up the standby Splunk UBA system
  NEXT
Failover to a standby Splunk UBA system

This documentation applies to the following versions of Splunk® User Behavior Analytics: 5.0.4, 5.0.4.1, 5.0.5, 5.0.5.1


Was this documentation topic helpful?

You must be logged into splunk.com in order to post comments. Log in now.

Please try to keep this discussion focused on the content covered in this documentation topic. If you have a more general question about Splunk functionality or are experiencing a difficulty with Splunk, consider posting a question to Splunkbase Answers.

0 out of 1000 Characters