Migrate an indexer cluster from single-site to multisite
You can migrate an indexer cluster from single-site to multisite. During this process, you incorporate your existing single-site cluster into a new multisite cluster.
Post-migration bucket behavior
After migration, all buckets created before migration continue to adhere to their single-site replication and search factor policies, by default. You can alter this behavior so that pre-migration buckets adhere instead to the multisite policies, by changing the
constrain_singlesite_buckets setting on the master's
server.conf to "false".
Buckets created after migration always adhere to the multisite policies.
Maintain existing buckets as single-site
In the default scenario, after the migration, the cluster holds both single-site and multisite buckets. It maintains them separately, following these rules:
- Single-site buckets (those existing at the time of migration) continue to respect the single-site
- Multisite buckets (those created after migration) follow the multisite
Convert existing buckets to multisite
If you configure the master node to convert existing buckets to multisite, all buckets will adhere to the settings for
site_search_factor, once the migration process is finished.
If you change the master node configuration before migration, existing buckets will follow the
site_search_factor policies immediately, post-migration.
You can also change the master node configuration later, post-migration. Any pre-migration buckets that have been following the single-site policies will then follow the multisite policies.
To see how many buckets will require conversion to multisite, use
services/cluster/master/buckets?filter=multisite_bucket=false&filter=standalone=false before changing the master node configuration.
Configure the master to convert existing buckets to multisite
To change the default behavior, so that existing single-site buckets adhere to the multisite replication and search factor policies,
constrain_singlesite_buckets setting on the master's
server.conf to "false":
[clustering] mode = master constrain_singlesite_buckets = false
You must restart the master node for the change to take effect.
Depending on when you want the associated, and potentially lengthy, bucket fixup to occur, you can make this configuration change either before migrating the cluster to muiltsite, or at some convenient time post-migration.
Perform the multisite migration
- The master node must be running Splunk Enterprise version 7.2 or later.
- All nodes in the post-migration cluster must adhere to the version compatibility rules described in Splunk Enterprise version compatibility. Therefore, before migrating to multisite, you might need to upgrade your single-site cluster. Follow the appropriate procedure in Upgrade an indexer cluster.
- If you want existing buckets to adhere to the multisite replication and search policies post-migration, you must change a configuration on the master node. See Configure the master to convert existing buckets to multisite. Alternatively, you can perform this step at any time post-migration.
To migrate a single-site cluster to multisite, configure each node for multisite:
1. Configure the master node for multisite and restart it, following the instructions in Configure multisite indexer clusters with the CLI. For example:
splunk edit cluster-config -mode master -multisite true -available_sites site1,site2 -site site1 -site_replication_factor origin:2,total:3 -site_search_factor origin:1,total:2 splunk restart
Note the following:
- Do not remove the existing single-site attributes for replication factor and search factor,
search_factor. The master needs them to handle the migrated buckets.
site_search_factormust be at least as large as
- If the number of peers on any site is less than the single-site
search_factor, you must reduce the values of those attributes to match the least number of peers on any site. For example, if
replication_factoris 3 and
search_factoris 2, and one of the sites has only 2 peers, you must change
replication_factorto 2. Otherwise, the migrated buckets might not meet the replication and search factors, due to the way the cluster replicates migrated buckets. See Multisite cluster does not meet its replication or search factors.
2. Set maintenance mode on the master:
splunk enable maintenance-mode
This step prevents unnecessary bucket fix-ups. See Use maintenance mode.
To confirm that the master has entered maintenance mode, run
splunk show maintenance-mode.
3. Configure the existing peer nodes for multisite. For each peer, specify its master node and site. For example:
splunk edit cluster-config -site site1
You will be prompted to restart the peer.
Do this for each peer, specifying the site for that peer.
4. If you want to add new peers to the cluster, follow the instructions in Configure multisite indexer clusters with the CLI. For example:
splunk edit cluster-config -mode slave -site site1 -master_uri https://10.160.31.200:8089 -replication_port 9887 splunk restart
Do this for each new peer that you want to add to the cluster.
5. Configure the search heads for multisite. For each search head, specify its master node and site. For example:
splunk edit cluster-master https://10.160.31.200:8089 -site site1
Do this for each search head, specifying the site for that search head.
Note: The configuration is essentially the same if the search heads are members of a search head cluster. See Integrate with a multisite indexer cluster in Distributed Search.
6. If you want to add new search heads to the cluster, follow the instructions in Configure multisite indexer clusters with the CLI. For example:
splunk edit cluster-config -mode searchhead -site site1 -master_uri https://10.160.31.200:8089 splunk restart
Do this for each new search head that you want to add to the cluster.
7. Disable maintenance mode on the master:
splunk disable maintenance-mode
To confirm that the master has left maintenance mode, run
splunk show maintenance-mode.
You can view the master dashboard to verify that all cluster nodes are up and running.
During the migration, the cluster tags each single-site bucket with a site value.
Note: You can also configure a multisite cluster by directly editing
server.conf. See Configure multisite indexer clusters with server.conf
8. If you are using indexer discovery to connect forwarders to the peer nodes, you must assign a site to each forwarder. See Use indexer discovery in a multisite cluster.
If you configured the master to convert existing single-site buckets to the multisite replication and search factor policies, bucket fixup will likely continue for some time after the cluster migration process itself completes. If you have a large number of existing buckets, the bucket fixup can continue for a long time.
How the cluster migrates and maintains existing buckets
Buckets in multisite clusters include a property that identifies the origin site. Buckets in single-site clusters do not include that property. So, when a cluster migrates from single-site to multisite, it must tag each single-site bucket with an origin site value. Since the bucket name includes the GUID of the originating peer, the cluster always knows the originating peer. With that information, it infers an origin site for the bucket:
- If the originating peer still exists in the cluster, the cluster assumes that the bucket originated on the site that the originating peer has been assigned to. It sets the bucket's origin to that site.
- If the originating peer is no longer in the cluster, the cluster assumes that the site with the most copies of the bucket is the origin site. It sets the bucket's origin to that site.
If the cluster is configured to maintain existing buckets as single-site
Here is how the cluster uses the inferred origin site to maintain the single-site bucket going forward, to handle any necessary fix-up so that the bucket continues to meet the single-site replication and search factors:
- If the cluster needs to replicate additional copies of the bucket to fulfill the replication factor, it only replicates within the bucket's inferred origin site.
- If the cluster needs to make a non-searchable copy of the bucket searchable to fulfill the search factor, it might do so on a non-origin site, if a non-searchable copy of that bucket already exists on some other site.
The cluster will never create a new copy of the bucket on a non-origin site.
Because, when pre-migraiton buckets are maintained as single-site, the cluster creates new copies of those buckets only on the origin site, it is possible to experience a situation where the replicaion factor cannot be met. For example, if
replication_factor is set to 3, but there are only two peer nodes on the origin site, the cluster will not create a third copy on a non-origin site, even if there had previouly been a third copy of the bucket on a non-origin site.
To remediate this situation, you can either convert pre-migration buckes to multisite or you can reduce the single-site replication factor so that it does not exceed the number of peer nodes on any site.
If the cluster is configured to convert existing buckets to multisite
The cluster uses the described methodology to infer an origin site for each bucket.
The cluster's process of converting buckets from adherence to the single-site policies to adherence to the multisite policies is the same as any bucket-fixup process, involving cross-site streaming and such. If you have a large number of existing buckets, the process can take a long time to complete. This fixup process has the same priority as any other concurrent fixup processes.
While fixup is continuing, the master dashboard indicates that the replication factor and search factors are not met. Once the fixup process is finished, the cluster returns to a complete state, as indicated by the master dashboard.
Handle blocked indexing on the new site
If you configure the master node for multisite clustering, but the new site is not yet fully operational, the master blocks indexing while it waits for enough peers to become available to fulfill the multisite replication factor. To unblock indexing, you can run the
splunk set indexing-ready command on the master. See Restart indexing in multisite cluster after master restart or site failure.
Configure the site search factor
View the master dashboard
This documentation applies to the following versions of Splunk® Enterprise: 7.2.0, 7.2.1, 7.2.2, 7.2.3, 7.2.4, 7.2.5, 7.2.6, 7.2.7, 7.2.8, 7.3.0, 7.3.1, 7.3.2