Decommission a site in a multisite indexer cluster

To decommission a site, you reconfigure several site-specific attributes.

Decommission a site

Caution: Before proceeding, be aware of these issues:

• If a peer on the decommissioned site contains any buckets that were created before the peer was part of a cluster, such buckets exist only on that peer and therefore will be lost when the site is decommissioned.
• Similarly, if the decommissioned site started out as a single-site cluster before it became part of a multisite cluster, any buckets that were created when it was a single-site cluster exist only on that site and will be lost when the site is decommissioned.
• Once the manager restarts at the end of the decommissioning process, it will commence bucket fix-up activities to return the cluster to a complete state. This can take a considerable amount of time, particularly if the decommissioned site held a large number of origin buckets.

Prerequsites

The cluster must meet these conditions before you decommission one of its sites:

• The cluster must be in a complete state.
• The manager node cannot be on the site that you are planning to decommission. If it is, move the manager to to a new site, following the guidelines in Handle manager node site failure.
• The site_replication_factor attribute must be configured so that at least one copy of each bucket resides on a site not due for decommissioning. For example, in a two-site cluster, a valid configuration would be site_replication_factor = origin:1,total:2.
• The site_search_factor attribute must be configured so that at least one searchable copy of each bucket resides on a site not due for decommissioning. For example, in a two-site cluster, a valid configuration would be site_search_factor = origin:1,total:2.
• If you need to reconfigure site_replication_factor or site_search_factor so that all buckets have copies on other sites, you must then wait until the manager completes fix-up activities and returns the cluster to a complete state before proceeding with the decommissioning.

Steps

1. For each search head on the site, either disable the search head or change the search head's site attribute to specify a remaining site. For example, to change the search head's site to site2:

   splunk edit cluster-manager https://10.160.31.200:8089  -site site2
   

See Configure the search heads.

2. For each forwarder that uses indexer discovery and specifies the decommissioned site, change its site attribute to specify a remaining site. For example, to change the forwarder's site to site2:

   [general]
   site = site2
   

   You must restart the forwarder for the configuration change to take effect. See Use indexer discovery in a multisite cluster.

3. Run splunk enable maintenance-mode on the manager. This step prevents unnecessary bucket fix-ups. See Use maintenance mode.
4. To confirm that the manager has entered maintenance mode, run splunk show maintenance-mode.
5. Update these attributes on the manager:
   • available_sites
   • site_replication_factor
   • site_search_factor
   • site_mappings
   For details on the necessary updates, see Reconfigure attributes.
6. Restart the manager. This step causes the attribute changes to take effect.
   Note: When the manager restarts, the peers from the decommissioned site will unsuccessfully try to rejoin the cluster. Ignore the resulting messages.
7. Run splunk disable maintenance-mode on the manager. This step starts fix-up activities for peers on the remaining sites.
8. To confirm that the manager has left maintenance mode, run splunk show maintenance-mode.
9. Run splunk stop on each peer on the decommissioned site. You can now remove these peers from the site.
10. To verify that that the decommissioning was successful, look at the top of the manager node dashboard. It should state that both the search factor and the replication factor are met. When both are met, the cluster is in a complete state, indicating that the decommissioning was successful. See View the manager node dashboard.
    Note: Because site decommissioning typically involves a large amount of bucket fix-up activity, it can take a considerable amount of time for the cluster to return to its complete state.

Reconfigure attributes

When you decommission a site, you must make changes to several site-specific attributes in server.conf on the manager node:

• available_sites: Remove the decommissioned site from the site list for this attribute.
• site_replication_factor and site_search_factor: If the decommissioned site is an explicit site in either of these attributes, remove it and otherwise reconfigure the attribute as necessary.
• site_mappings: Add a mapping for the decommissioned site to this attribute. See Map the decommissioned site.

You must restart the manager node after you change any of these attributes.

Map the decommissioned site

When a site gets decommissioned, the origin bucket copies for that site remain bound to the decommissioned site until you map the site to a remaining, active site. This can make the cluster unable to meet its replication or search factors.

To deal with this issue, you can map decommissioned sites to active sites. The bucket copies for which a decommissioned site is the origin site will then be replicated to the active site specified by the mapping, allowing the cluster to again meet its replication and search factors.

Note: The site_replication_factor and site_search_factor attributes determine the cluster's number of origin bucket copies.

Syntax

Map the site before you decommission it. See Decommission a site.

To map a site that you are planning to decommission to a remaining site, use the site_mappings attribute in server.conf. Set this attribute on the manager node only, using this syntax:

site_mappings = <comma-separated string>

Note the following:

• The <comma-separated string> contains mappings from decommissioned sites to the remaining, active sites. These mappings can be of two types:
  • <decommissioned_site_id>:<active_site_id>. For example, site2:site3, where site2 is a decommissioned site and site3 is an active site. This is called an explicit mapping. There can be multiple explicit mappings.
  • default_mapping:<active_site_id>. For example, default_mapping:site4, where site4 is an active site. There can be at most one default mapping. It is recommended that you always include a default mapping, as fallback for an incorrect or missing explicit mapping.
• In the case of <decommissioned_site_id>:<active_site_id>, the origin bucket copies in <decommissioned_site_id> will get replicated from remaining sites to peers on the <active_site_id>. This allows the cluster to meet the requirements of its replication and search factors.
• In the case of default_mapping:<active_site_id>, the origin bucket copies for any decommissioned site without an explicit mapping will get replicated to <active_site_id>.
• If an active site in a mapping is itself later decommissioned, its previous mappings must be remapped to a currently active site. For example, in the case of site2:site3, if site3 is itself decommissioned, you must replace the previous mapping, site2:site3, with a new set of mappings that map both site2 and site3 to an active site, such as site4, using the string site2:site4,site3:site4.

Restart the manager node after changing this attribute.

Examples

These examples assume a cluster that originally had five sites, site1 through site5.

• "site_mappings = site2:site3" This configuration maps a decommissioned site2 to an active site3. The mapping causes origin bucket copies for site2 to replicate to site3. There is no default site mapping.
• "site_mappings = site1:site3,default_mapping:site4" This configuration maps a decommissioned site1 to site3 and maps all other decommissioned sites to site4. The mapping causes origin bucket copies for the decommissioned sites to replicate to their respective mapped sites.
• "site_mappings = default_mapping:site5" This configuration maps all decommissioned sites to site5. The mapping causes origin bucket copies for all decommissioned sites to replicate to site5.