Migrate an indexer cluster from single-site to multisite

You can migrate an indexer cluster from single-site to multisite. During this process, you incorporate your existing single-site cluster into a new multisite cluster.

Post-migration bucket behavior

After migration, all buckets created before migration continue to adhere to their single-site replication and search factor policies by default. You can change this behavior so that legacy buckets adhere instead to the multisite policies.

Buckets created after migration always adhere to the multisite policies.

Maintain legacy buckets as single-site

By default, after migration, the cluster holds both single-site and multisite buckets. It maintains them separately, following these rules:

• Single-site legacy buckets (those existing at the time of migration) continue to respect the single-site replication_factor and search_factor settings.
• Multisite buckets (those created after migration) follow the multisite site_replication_factor and site_search_factor policies.

Convert legacy buckets to multisite

You can configure the master node to convert legacy buckets to multisite. This process causes buckets that were following the single-site replication and search policies, pre-migration, to adhere instead to the multisite replication and search policies.

When deciding whether to convert legacy buckets, you must weigh the value of maintaining those buckets across multiple sites against the possibly considerable time sink required for the bucket fixup activity needed to convert them to muitisite.

You can make the necessary configuration change either before migration or at any point after migration.

If you change the configuration before migration, legacy buckets will follow the site_replication_factor and site_search_factor policies immediately post-migration.

If you change the configuration post-migration, any pre-migration buckets that have been following the single-site policies will then follow the multisite policies.

To see how many buckets will require conversion to multisite, use services/cluster/master/buckets?filter=multisite_bucket=false&filter=standalone=false before changing the master node configuration.

Configure the master to convert legacy buckets to multisite

To cause legacy single-site buckets to adhere to the multisite replication and search factor policies, change the constrain_singlesite_buckets setting in the master's server.conf file to "false":

[clustering]
mode = master
constrain_singlesite_buckets = false

You must restart the master node for the change to take effect.

Perform the multisite migration

Prerequisites

• The master node must be running Splunk Enterprise version 7.2 or later.
• All nodes in the post-migration cluster must adhere to the version compatibility rules described in Splunk Enterprise version compatibility. Therefore, before migrating to multisite, you might need to upgrade your single-site cluster. Follow the appropriate procedure in Upgrade an indexer cluster.
• If you want existing buckets to adhere to the multisite replication and search policies post-migration, you must change a configuration on the master node. See Configure the master to convert existing buckets to multisite. Alternatively, you can perform this step at any time post-migration.

Steps

To migrate a single-site cluster to multisite, configure each node for multisite:

1. Configure the master node for multisite and restart it, following the instructions in Configure multisite indexer clusters with the CLI. For example:

splunk edit cluster-config -mode master -multisite true -available_sites site1,site2 -site site1 -site_replication_factor origin:2,total:3 -site_search_factor origin:1,total:2

splunk restart

Note the following:

• Do not remove the existing single-site attributes for replication factor and search factor, replication_factor and search_factor. The master needs them to handle the migrated buckets.

• The total values for site_replication_factor and site_search_factor must be at least as large as replication_factor and search_factor, respectively.

• If the number of peers on any site is less than the single-site replication_factor or search_factor, you must reduce the values of those attributes to match the least number of peers on any site. For example, if replication_factor is 3 and search_factor is 2, and one of the sites has only 2 peers, you must change replication_factor to 2. Otherwise, the migrated buckets might not meet the replication and search factors, due to the way the cluster replicates migrated buckets. See Multisite cluster does not meet its replication or search factors.

2. Set maintenance mode on the master:

splunk enable maintenance-mode

This step prevents unnecessary bucket fix-ups. See Use maintenance mode.

To confirm that the master has entered maintenance mode, run splunk show maintenance-mode.

3. Configure the existing peer nodes for multisite. For each peer, specify its master node and site. For example:

splunk edit cluster-config -site site1 

You will be prompted to restart the peer.

Do this for each peer, specifying the site for that peer.

4. If you want to add new peers to the cluster, follow the instructions in Configure multisite indexer clusters with the CLI. For example:

splunk edit cluster-config -mode slave -site site1 -master_uri https://10.160.31.200:8089 -replication_port 9887 

splunk restart

Do this for each new peer that you want to add to the cluster.

5. Configure the search heads for multisite. For each search head, specify its master node and site. For example:

splunk edit cluster-master https://10.160.31.200:8089 -site site1 

Do this for each search head, specifying the site for that search head.

Note: The configuration is essentially the same if the search heads are members of a search head cluster. See Integrate with a multisite indexer cluster in Distributed Search.

6. If you want to add new search heads to the cluster, follow the instructions in Configure multisite indexer clusters with the CLI. For example:

splunk edit cluster-config -mode searchhead -site site1 -master_uri https://10.160.31.200:8089  

splunk restart

Do this for each new search head that you want to add to the cluster.

7. Disable maintenance mode on the master:

splunk disable maintenance-mode

To confirm that the master has left maintenance mode, run splunk show maintenance-mode.

You can view the master dashboard to verify that all cluster nodes are up and running.

During the migration, the cluster tags each single-site bucket with a site value.

Note: You can also configure a multisite cluster by directly editing server.conf. See Configure multisite indexer clusters with server.conf

8. If you are using indexer discovery to connect forwarders to the peer nodes, you must assign a site to each forwarder. See Use indexer discovery in a multisite cluster.

If you configured the master to convert existing single-site buckets to the multisite replication and search factor policies, bucket fixup will likely continue for some time after the cluster migration process itself completes. If you have a large number of existing buckets, the bucket fixup can continue for a long time.

How the cluster migrates and maintains existing buckets

Buckets in multisite clusters include a property that identifies the origin site. Buckets in single-site clusters do not include that property. So, when a cluster migrates from single-site to multisite, it must tag each single-site bucket with an origin site value. Since the bucket name includes the GUID of the originating peer, the cluster always knows the originating peer. With that information, it infers an origin site for the bucket:

• If the originating peer still exists in the cluster, the cluster assumes that the bucket originated on the site that the originating peer has been assigned to. It sets the bucket's origin to that site.
• If the originating peer is no longer in the cluster, the cluster assumes that the site with the most copies of the bucket is the origin site. It sets the bucket's origin to that site.

If the cluster is configured to maintain existing buckets as single-site

Here is how the cluster uses the inferred origin site to maintain the single-site bucket going forward, to handle any necessary fix-up so that the bucket continues to meet the single-site replication and search factors:

• If the cluster needs to replicate additional copies of the bucket to fulfill the replication factor, it only replicates within the bucket's inferred origin site.
• If the cluster needs to make a non-searchable copy of the bucket searchable to fulfill the search factor, it might do so on a non-origin site, if a non-searchable copy of that bucket already exists on some other site.

The cluster will never create a new copy of the bucket on a non-origin site.

Because, when pre-migration buckets are maintained as single-site, the cluster creates new copies of those buckets only on the origin site, it is possible to experience a situation where the replication factor cannot be met. For example, if replication_factor is set to 3, but there are only two peer nodes on the origin site, the cluster will not create a third copy on a non-origin site, even if there had previouly been a third copy of the bucket on a non-origin site.

To remediate this situation, you can either convert pre-migration buckes to multisite or you can reduce the single-site replication factor so that it does not exceed the number of peer nodes on any site.

If the cluster is configured to convert existing buckets to multisite

The cluster uses the described methodology to infer an origin site for each bucket.

The cluster's process of converting buckets from adherence to the single-site policies to adherence to the multisite policies is the same as any bucket-fixup process, involving cross-site streaming and such. If you have a large number of existing buckets, the process can take a long time to complete. This fixup process has the same priority as any other concurrent fixup processes.

While fixup is continuing, the master dashboard indicates that the replication factor and search factors are not met. Once the fixup process is finished, the cluster returns to a complete state, as indicated by the master dashboard.

Handle blocked indexing on the new site

If you configure the master node for multisite clustering, but the new site is not yet fully operational, the master blocks indexing while it waits for enough peers to become available to fulfill the multisite replication factor. To unblock indexing, you can run the splunk set indexing-ready command on the master. See Restart indexing in multisite cluster after master restart or site failure.