Bootstrap SmartStore indexes

Bootstrapping is the process of transferring SmartStore indexes to a new indexer cluster or standalone indexer. You first bring down an old indexer cluster or indexer that has SmartStore indexes. You then point a new indexer cluster or standalone indexer to the remote storage location previously used by the SmartStore indexes on the decommissioned cluster or indexer. The new cluster or indexer inherits the SmartStore buckets from the old cluster or indexer.

For example, if you want to replace an indexer cluster, you can take down the old cluster and bootstrap a new cluster. The new cluster inherits all the buckets from the old cluster. This can be useful if you need to upgrade the machines running your indexer cluster.

Similarly, you can use bootstrapping for scalability and high availability purposes, to move SmartStore indexes from a standalone indexer to an indexer cluster.

Multiple indexer clusters or standalone indexers cannot access the same remote volume. In other words, the value of any path setting in indexes.conf must be unique to a single running indexer or cluster. Do not share path settings among multiple indexers or clusters.

Therefore, you must shut down the cluster or standalone indexer currently accessing buckets from a particular remote volume before you bootstrap a new cluster or indexer to access buckets from the same remote volume.

Types of bootstrapping

You can bootstrap SmartStore indexes from one indexer cluster or standalone indexer to another indexer cluster or standalone indexer. In other words, these types of bootstrapping are supported:

• Indexer cluster to indexer cluster
• Standalone indexer to standalone indexer
• Standalone indexer to indexer cluster
• Indexer cluster to standalone indexer

Bootstrap indexes onto an indexer cluster

You can boostrap indexes onto an indexer cluster from an old cluster or standalone indexer.

How to bootstrap indexes at initial start-up

You can bootstrap all indexes, or just a subset of indexes. See Deploy SmartStore on a new indexer cluster for information on how to use indexes.conf to configure SmartStore either globally or on a per-index basis.

You can bootstrap SmartStore indexes when the cluster initially starts up or later in its life cycle. This section describes how to bootstrap indexes at initial start-up.

To bootstrap SmartStore indexes onto a new indexer cluster:

1. Bring down the old cluster or standalone indexer after ensuring that all local data, including data from any hot buckets, is available on the remote store.
2. Bring up the new cluster and configure its indexes for SmartStore. Follow the procedure in Deploy SmartStore on an indexer cluster.
   Note the following:
   • When configuring indexes.conf, add stanzas for all SmartStore indexes that you want to bootstrap.
   • Confirm that you have set repFactor=auto globally, so that it applies to all indexes in the cluster, including the ones that you intend to bootstrap.
   • Configure the path attribute to point to the same remote storage location as on the old cluster or standalone indexer.
   • Maintain the remote store encryption and access settings used by the old cluster or standalone indexer.
   • All indexes that you bootstrap must be SmartStore-enabled on the new cluster. You cannot use bootstrapping to revert an index to non-SmartStore.

When bootstrapping occurs

The master node initiates bootstrapping under either of these conditions:

• When the replication factor number of peer nodes have registered with the master after indexer cluster start-up.
• After the master pushes a new bootstrapping configuration to the peer nodes through the configuration bundle push method. This allows you to bootstrap SmartStore indexes after the cluster is already up and running, or to bootstrap SmartStore indexes selectively at varying points in the life of the cluster.

The bootstrap process

During bootstrapping, the master node coordinates a discovery process:

1. The master obtains a list of all SmartStore indexes from the peer nodes.
2. Bootstrapping proceeds on a per-index basis:
   1. The master assigns one peer to obtain a list of all buckets for the index from the remote store.
   2. The peer returns the bucket list to the master in batches.

      Batch size is controlled by the recreate_index_fetch_bucket_batch_size attribute in the master node's server.conf file.

   3. The master uses the list to assign buckets, randomly but evenly, across all available peer nodes.
   4. The master provides each peer node with a set of target peer nodes, of a number equal to the replication factor.
   5. Each peer node downloads, from the remote store, the metadata for its assigned buckets.
   6. The peer node uses the downloaded metadata to update the index's .bucketManifest file and to create empty directories for its assigned buckets. The primary marker for each bucket resides on the peer that downloaded that bucket's metadata from the remote store.
   7. The peer node replicates copies of its primary buckets' metadata to target peer nodes to meet the replication factor.

The bootstrapping process proceeds quickly because only the bucket metadata is downloaded and replicated. The bucket contents themselves are not downloaded during bootstrapping, but only later, if needed for searching.

You can use the monitoring console to monitor bootstrapping progress. See Troubleshoot with the monitoring console.

Bootstrap indexes onto a standalone indexer

You can bootstrap indexes onto a standalone indexer from an old standalone indexer or cluster.

You cannot run multiple standalone indexers against the same remote store. Only a single standalone indexer can access a particular remote store.

How to bootstrap indexes at initial start-up

You can bootstrap all indexes, or just a subset of indexes, from the remote store. See Deploy SmartStore on a new standalone indexer for information on how to use indexes.conf to configure SmartStore either globally or on a per-index basis.

You can bootstrap SmartStore indexes when the indexer initially starts up or later in its life cycle. This section describes how to bootstrap indexes at initial start-up. The process for bootstrapping indexes later is a simple variant of adding an index to an existing indexer.

To bootstrap SmartStore indexes onto a new indexer:

1. Bring down the old standalone indexer or cluster after ensuring that all local data, including data from any hot buckets, is available on the remote store.
2. Bring up the new indexer and configure its indexes for SmartStore. Follow the procedure in Deploy SmartStore on a new standalone indexer.
   Note the following:
   • When configuring indexes.conf, add stanzas for all SmartStore indexes that you want to bootstrap.
   • Configure the path attribute to point to the same remote storage location as on the old cluster or standalone indexer.
   • Maintain the remote store encryption and access settings used by the old cluster or standalone indexer.
   • All indexes that you bootstrap must be SmartStore-enabled on the new indexer. You cannot use bootstrapping to revert an index to non-SmartStore.
3. Restart the indexer.

The indexer initiates bootstrapping after it restarts.

The bootstrap process

During bootstrapping, the indexer performs a discovery process. Bootstrapping proceeds on a per-index basis:

1. The indexer obtains a list of all buckets for the index from the remote store.
2. The indexer downloads, from the remote store, the metadata for the index's buckets.
3. The indexer uses the downloaded metadata to update the index's .bucketManifest file and to create empty directories for the buckets.

The bootstrapping process proceeds quickly because only the bucket metadata is downloaded. The bucket contents themselves are not downloaded during bootstrapping, but only later, if needed for searching.

You can use the monitoring console to monitor bootstrapping progress. See Troubleshoot with the monitoring console.