Migrate an indexer cluster from single-site to multisite

You can migrate an indexer cluster from single-site to multisite. After the migration, the cluster holds both single-site and multi-site buckets. It maintains them separately, following these rules:

• Single-site buckets (those existing at the time of migration) continue to respect their single-site replication and search factors. You cannot convert them to multisite.
• Multisite buckets (those created after migration) follow the multisite replication and search factor policies.

Perform the multisite migration

Important: The migration process does not alter the version of Splunk Enterprise that the instances are running on. To migrate to a multisite cluster, the instances must be running version 6.1 or higher. Therefore, before migrating to multisite, you might need to upgrade your single-site cluster. Follow the appropriate procedure in Upgrade an indexer cluster.

To migrate a single-site cluster to multisite, configure each node for multisite:

1. Configure the master node for multisite and restart it, following the instructions in Configure multisite indexer clusters with the CLI. For example:

splunk edit cluster-config -mode master -multisite true -available_sites site1,site2 -site site1 -site_replication_factor origin:2,total:3 -site_search_factor origin:1,total:2

splunk restart

Note the following:

• Do not remove the existing single-site attributes for replication factor and search factor, replication_factor and search_factor. The master needs them to handle the migrated buckets.

• The total values for site_replication_factor and site_search_factor must be at least as large as replication_factor and search_factor, respectively.

• If the number of peers on any site is less than the single-site replication_factor or search_factor, you must reduce the values of those attributes to match the least number of peers on any site. For example, if replication_factor is 3 and search_factor is 2, and one of the sites has only 2 peers, you must change replication_factor to 2. Otherwise, the migrated buckets might not meet the replication and search factors, due to the way the cluster replicates migrated buckets. See Multisite cluster does not meet its replication or search factors.

2. Set maintenance mode on the master:

splunk enable maintenance-mode

This step prevents unnecessary bucket fix-ups. See Use maintenance mode.

To confirm that the master has entered maintenance mode, run splunk show maintenance-mode.

3. Configure the existing peer nodes for multisite. For each peer, specify its master node and site. For example:

splunk edit cluster-config -site site1 

You will be prompted to restart the peer.

Do this for each peer, specifying the site for that peer.

4. If you want to add new peers to the cluster, follow the instructions in Configure multisite indexer clusters with the CLI. For example:

splunk edit cluster-config -mode slave -site site1 -master_uri https://10.160.31.200:8089 -replication_port 9887 

splunk restart

Do this for each new peer that you want to add to the cluster.

5. Configure the search heads for multisite. For each search head, specify its master node and site. For example:

splunk edit cluster-master https://10.160.31.200:8089 -site site1 

Do this for each search head, specifying the site for that search head.

Note: The configuration is essentially the same if the search heads are members of a search head cluster. See Integrate with a multisite indexer cluster in Distributed Search.

6. If you want to add new search heads to the cluster, follow the instructions in Configure multisite indexer clusters with the CLI. For example:

splunk edit cluster-config -mode searchhead -site site1 -master_uri https://10.160.31.200:8089  

splunk restart

Do this for each new search head that you want to add to the cluster.

7. Disable maintenance mode on the master:

splunk disable maintenance-mode

To confirm that the master has left maintenance mode, run splunk show maintenance-mode.

You can view the master dashboard to verify that all cluster nodes are up and running.

During the migration, the cluster tags each single-site bucket with a site value.

Note: You can also configure a multisite cluster by directly editing server.conf. See Configure multisite indexer clusters with server.conf

8. If you are using indexer discovery to connect forwarders to the peer nodes, you must assign a site to each forwarder. See Use indexer discovery in a multisite cluster.

How the cluster migrates and maintains single-site buckets

Buckets in multisite clusters include a property that identifies the origin site. Buckets in single-site clusters do not include that property. So, when a cluster migrates from single-site to multisite, it must tag each single-site bucket with an origin site value. Since the bucket name includes the GUID of the originating peer, the cluster always knows the originating peer. With that information, it infers an origin site for the bucket:

• If the originating peer still exists in the cluster, the cluster assumes that the bucket originated on the site that the originating peer has been assigned to. It sets the bucket's origin to that site.
• If the originating peer is no longer in the cluster, the cluster assumes that the site with the most copies of the bucket is the origin site. It sets the bucket's origin to that site.

Here is how the cluster uses the inferred origin site to maintain the single-site bucket going forward, to handle any necessary fix-up so that the bucket continues to meet the single-site replication and search factors:

• If the cluster needs to replicate additional copies of the bucket to fulfill the replication factor, it only replicates within the bucket's inferred origin site.
• If the cluster needs to make a non-searchable copy of the bucket searchable to fulfill the search factor, it might do so on a non-origin site, if a non-searchable copy of that bucket already exists on some other site.

The cluster will never create a new copy of the bucket on a non-origin site.

Handle problems with the new site

If you configure the master node for multisite clustering, but the new site is not yet fully operational, the master blocks indexing while it waits for enough peers to become available to fulfill the multisite replication factor. To unblock indexing, you can run the splunk set indexing-ready command on the master. See Restart indexing in multisite cluster after master restart or site failure.