Rebalance the cluster primary buckets
When you restart a master or peer node, the master rebalances the set of primary bucket copies across the peers, in an attempt to spread the primary copies as equitably as possible. Ideally, if you have four peers and 300 buckets, each peer would hold 75 primary copies. The purpose of rebalancing is to even the search load across the set of peers.
To achieve rebalancing, the master reassigns the primary state from existing bucket copies to searchable copies of the same buckets on other peers, as necessary. This rebalancing is a best-effort attempt; there is no guarantee that full, perfect rebalance will result.
Rebalancing occurs automatically whenever a peer or master node joins (or rejoins) the cluster.
Note: Even though rebalancing occurs when a new peer joins the cluster, that peer won't participate in the rebalancing, because it does not yet have any bucket copies. The rebalancing takes place among any existing peers that have searchable bucket copies.
If you want to initiate the process manually, you can either restart a peer or hit the
/services/cluster/master/control/control/rebalance_primaries REST endpoint on the master. For example, run this command on the master node:
curl -k -u admin:pass --request POST \ https://localhost:8089/services/cluster/master/control/control/rebalance_primaries
For more information, refer to the REST API documentation for cluster/master/control.
Note: When rebalancing a bucket, the master just reassigns the primary state from one searchable copy to another searchable copy of the same bucket, if there is one and if, by doing so, the balance of primaries across peers will be improved. It does not cause peers to stream bucket copies, and it does not cause peers to make unsearchable copies searchable. So, for example, if you add a new peer that does not currently have any bucket copies, rebalancing will not cause the peer to gain any copies. Similarly, if an existing peer does not have any searchable copies, it will not gain any primaries during rebalancing.
In summary, cluster rebalancing occurs under these circumstances:
- A peer joins or rejoins a cluster.
- A master comes up (after going down).
- You manually hit the
rebalance_primariesREST endpoint on the master.
Restart the entire cluster or a single peer node
Remove excess bucket copies
This documentation applies to the following versions of Splunk® Enterprise: 6.0, 6.0.1, 6.0.2, 6.0.3, 6.0.4, 6.0.5, 6.0.6, 6.0.7, 6.0.8, 6.0.9, 6.0.10, 6.0.11, 6.0.12, 6.0.13, 6.0.14, 6.0.15