Rebalance the indexer cluster primary bucket copies

When you start or restart a master or peer node, the master rebalances the set of primary bucket copies across the peers, in an attempt to spread the primary copies as equitably as possible. Ideally, if you have four peers and 300 buckets, each peer would hold 75 primary copies. The purpose of rebalancing is to even the search load across the set of peers.

How rebalancing works

To achieve rebalancing, the master reassigns the primary state from existing bucket copies to searchable copies of the same buckets on other peers, as necessary. This rebalancing is a best-effort attempt; there is no guarantee that full, perfect rebalance will result.

Rebalancing occurs automatically whenever a peer or master node joins or rejoins the cluster. In the case of a rolling restart, rebalancing occurs once, at the end of the process.

Note: Even though rebalancing occurs when a new peer joins the cluster, that peer won't participate in the rebalancing, because it does not yet have any bucket copies. The rebalancing takes place among any existing peers that have searchable bucket copies.

When rebalancing a bucket, the master just reassigns the primary state from one searchable copy to another searchable copy of the same bucket, if there is one and if, by doing so, the balance of primaries across peers will improve. It does not cause peers to stream bucket copies, and it does not cause peers to make unsearchable copies searchable. If an existing peer does not have any searchable copies, it will not gain any primaries during rebalancing.

Initiate rebalancing manually

If you want to initiate the process manually, you can either restart a peer or hit the /services/cluster/master/control/control/rebalance_primaries REST endpoint on the master. For example, run this command on the master node:

curl -k -u admin:pass --request POST \
  https://localhost:8089/services/cluster/master/control/control/rebalance_primaries

For more information, refer to the REST API documentation for cluster/master/control.

Rebalance a multisite cluster

There are a few differences in how rebalancing works in a multisite cluster. In a multisite cluster, multiple sites typically have full sets of primary copies. When you rebalance the cluster, the rebalancing occurs independently for each site. For example, in a two-site cluster, the cluster separately rebalances the primaries in site1 and site2. It does not shift primaries between the two sites.

The start or restart of a peer on any site triggers rebalancing on all sites. For example, if you restart a peer on site1 in a two-site cluster, rebalancing occurs on both site1 and site2.

Summary

Cluster rebalancing is the rebalancing of the primary assignments across existing searchable copies in the cluster.

It occurs under these circumstances:

• A peer joins or rejoins a cluster.
• At the end of a rolling restart.
• A master rejoins the cluster.
• You manually hit the rebalance_primaries REST endpoint on the master.