Bootstrap SmartStore indexes onto an indexer cluster

Bootstrapping is the process of transferring SmartStore buckets to a new indexer cluster. You first bring down an old indexer cluster with SmartStore indexes. You then point a new indexer cluster to the remote storage location previously used by the decommissioned cluster. The new cluster inherits the SmartStore buckets from the old cluster.

For example, if you want to replace an indexer cluster, you can take down the old cluster and bootstrap a new cluster. The new cluster inherits all the buckets from the old cluster. This can be useful if you need to upgrade the machines running your indexer cluster.

You cannot run two indexer clusters against the same remote store. That is, you must shut down the cluster currently accessing buckets from the remote store before you bootstrap a new cluster to access the same buckets from the remote store.

How to bootstrap indexes onto a cluster at initial start-up

You can bootstrap all indexes, or just a subset of indexes, from the remote store. See Deploy SmartStore on a new indexer cluster for information on how to use indexes.conf to configure SmartStore either globally or on a per-index basis.

You can bootstrap SmartStore indexes when the cluster initially starts up or later in its life cycle. This section describes how to bootstrap indexes at cluster initial start-up. The process for bootstrapping indexes later is a simple variant of adding an index to an existing indexer cluster.

To bootstrap SmartStore indexes onto a new indexer cluster:

1. Bring down the old cluster after ensuring that all local data, including data from any hot buckets, is available on the remote store.
2. Bring up the new cluster and configure its indexes for SmartStore. Follow the procedure in Deploy SmartStore on an indexer cluster.
   Note the following:
   • When configuring indexes.conf, include all indexes that you want to bootstrap onto the new cluster.
   • You must set repFactor=auto on all indexes that you intend to bootstrap.
   • Configure the path attribute to point to the same remote storage location as on the old cluster.
   • You must maintain the remote store encryption and access settings used by the old cluster.
   • All indexes that you bootstrap must be SmartStore-enabled on the new cluster. You cannot use bootstrapping to revert an index to non-SmartStore.

When the indexer cluster master node initiates the bootstrap process

The master node initiates bootstrapping under either of these conditions:

• When the replication factor number of peer nodes have registered with the master after indexer cluster start-up.
• After the master pushes a new bootstrapping configuration to the peer nodes through the configuration bundle method. This allows you to bootstrap SmartStore indexes after the cluster is already up and running, or to bootstrap SmartStore indexes selectively at varying points in the life of the cluster.

The bootstrap process

During bootstrapping, the master node coordinates a discovery process:

1. The master obtains a list of all SmartStore indexes from the peer nodes.
2. Bootstrapping proceeds on a per-index basis:
   1. The master assigns one peer to obtain a list of all buckets for the index from the remote store.
   2. The peer returns the bucket list to the master in batches.

      Batch size is controlled by the recreate_index_fetch_bucket_batch_size attribute in the master node's server.conf file.

   3. The master uses the list to assign buckets, randomly but evenly, across all available peer nodes.
   4. The master provides each peer node with a set of target peer nodes, of a number equal to the replication factor.
   5. Each peer node downloads, from the remote store, the metadata for its assigned buckets.
   6. The peer node uses the downloaded metadata to update the index's .bucketManifest file and to create empty directories for its assigned buckets. The primary marker for each bucket resides on the peer that downloaded that bucket's metadata from the remote store.
   7. The peer node replicates copies of its primary buckets' metadata to target peer nodes to meet the replication factor.

The bootstrapping process proceeds quickly because only the bucket metadata is downloaded and replicated. The bucket contents themselves are not downloaded during bootstrapping, but only later, if needed for searching.