Decommission a site in a multisite indexer cluster
To decommission a site, you reconfigure several site-specific attributes.
Decommission a site
Caution: Before proceeding, be aware of these issues:
- If a peer on the decommissioned site contains any buckets that were created before the peer was part of a cluster, such buckets exist only on that peer and therefore will be lost when the site is decommissioned.
- Similarly, if the decommissioned site started out as a single-site cluster before it became part of a multisite cluster, any buckets that were created when it was a single-site cluster exist only on that site and will be lost when the site is decommissioned.
- Once the master restarts at the end of the decommissioning process, it will commence bucket fix-up activities to return the cluster to a complete state. This can take a considerable amount of time, particularly if the decommissioned site held a large number of origin buckets.
Prerequsites
The cluster must meet these conditions before you decommission one of its sites:
- The cluster must be in a complete state.
- The master node cannot be on the site that you are planning to decommission. If it is, move the master to to a new site, following the guidelines in Handle master site failure.
- The
site_replication_factor
attribute must be configured so that at least one copy of each bucket resides on a site not due for decommissioning. For example, in a two-site cluster, a valid configuration would besite_replication_factor = origin:1,total:2
. - The
site_search_factor
attribute must be configured so that at least one searchable copy of each bucket resides on a site not due for decommissioning. For example, in a two-site cluster, a valid configuration would besite_search_factor = origin:1,total:2
. - If you need to reconfigure
site_replication_factor
orsite_search_factor
so that all buckets have copies on other sites, you must then wait until the master completes fix-up activities and returns the cluster to a complete state before proceeding with the decommissioning.
Steps
- For each search head on the site, either disable the search head or change the search head's
site
attribute to specify a remaining site. For example, to change the search head's site to site2:splunk edit cluster-master https://10.160.31.200:8089 -site site2
- For each forwarder that uses indexer discovery and specifies the decommissioned site, change its
site
attribute to specify a remaining site. For example, to change the forwarder's site to site2:[general] site = site2
You must restart the forwarder for the configuration change to take effect. See Use indexer discovery in a multisite cluster.
- Run
splunk enable maintenance-mode
on the master. This step prevents unnecessary bucket fix-ups. See Use maintenance mode. - To confirm that the master has entered maintenance mode, run
splunk show maintenance-mode
. - Update these attributes on the master:
available_sites
site_replication_factor
site_search_factor
site_mappings
- Restart the master. This step causes the attribute changes to take effect.
Note: When the master restarts, the peers from the decommissioned site will unsuccessfully try to rejoin the cluster. Ignore the resulting messages. -
Run
splunk disable maintenance-mode
on the master. This step starts fix-up activities for peers on the remaining sites. - To confirm that the master has left maintenance mode, run
splunk show maintenance-mode
. - Run
splunk stop
on each peer on the decommissioned site. You can now remove these peers from the site. -
To verify that that the decommissioning was successful, look at the top of the master dashboard. It should state that both the search factor and the replication factor are met. When both are met, the cluster is in a complete state, indicating that the decommissioning was successful. See View the master dashboard.
Note: Because site decommissioning typically involves a large amount of bucket fix-up activity, it can take a considerable amount of time for the cluster to return to its complete state.
See Configure the search heads.
Reconfigure attributes
When you decommission a site, you must make changes to several site-specific attributes in server.conf
on the master node:
available_sites
: Remove the decommissioned site from the site list for this attribute.site_replication_factor
andsite_search_factor
: If the decommissioned site is an explicit site in either of these attributes, remove it and otherwise reconfigure the attribute as necessary.site_mappings
: Add a mapping for the decommissioned site to this attribute. See Map the decommissioned site.
You must restart the master node after you change any of these attributes.
Map the decommissioned site
When a site gets decommissioned, the origin bucket copies for that site remain bound to the decommissioned site until you map the site to a remaining, active site. This can make the cluster unable to meet its replication or search factors.
To deal with this issue, you can map decommissioned sites to active sites. The bucket copies for which a decommissioned site is the origin site will then be replicated to the active site specified by the mapping, allowing the cluster to again meet its replication and search factors.
Note: The site_replication_factor
and site_search_factor
attributes determine the cluster's number of origin bucket copies.
Syntax
Map the site before you decommission it. See Decommission a site.
To map a site that you are planning to decommission to a remaining site, use the site_mappings
attribute in server.conf
. Set this attribute on the master node only, using this syntax:
site_mappings = <comma-separated string>
Note the following:
- The
<comma-separated string>
contains mappings from decommissioned sites to the remaining, active sites. These mappings can be of two types:<decommissioned_site_id>:<active_site_id>
. For example, site2:site3, where site2 is a decommissioned site and site3 is an active site. This is called an explicit mapping. There can be multiple explicit mappings.default_mapping:<active_site_id>
. For example, default_mapping:site4, where site4 is an active site. There can be at most one default mapping. It is recommended that you always include a default mapping, as fallback for an incorrect or missing explicit mapping.
- In the case of
<decommissioned_site_id>:<active_site_id>
, the origin bucket copies in<decommissioned_site_id>
will get replicated from remaining sites to peers on the<active_site_id>
. This allows the cluster to meet the requirements of its replication and search factors. - In the case of
default_mapping:<active_site_id>
, the origin bucket copies for any decommissioned site without an explicit mapping will get replicated to<active_site_id>
. - If an active site in a mapping is itself later decommissioned, its previous mappings must be remapped to a currently active site. For example, in the case of site2:site3, if site3 is itself decommissioned, you must replace the previous mapping, site2:site3, with a new set of mappings that map both site2 and site3 to an active site, such as site4, using the string site2:site4,site3:site4.
Restart the master node after changing this attribute.
Examples
These examples assume a cluster that originally had five sites, site1 through site5.
- "site_mappings = site2:site3" This configuration maps a decommissioned site2 to an active site3. The mapping causes origin bucket copies for site2 to replicate to site3. There is no default site mapping.
- "site_mappings = site1:site3,default_mapping:site4" This configuration maps a decommissioned site1 to site3 and maps all other decommissioned sites to site4. The mapping causes origin bucket copies for the decommissioned sites to replicate to their respective mapped sites.
- "site_mappings = default_mapping:site5" This configuration maps all decommissioned sites to site5. The mapping causes origin bucket copies for all decommissioned sites to replicate to site5.
Move a peer to a new site | Basic indexer cluster concepts for advanced users |
This documentation applies to the following versions of Splunk® Enterprise: 7.0.0, 7.0.1, 7.0.2, 7.0.3, 7.0.4, 7.0.5, 7.0.6, 7.0.7, 7.0.8, 7.0.9, 7.0.10, 7.0.11, 7.0.13, 7.1.0, 7.1.1, 7.1.2, 7.1.3, 7.1.4, 7.1.5, 7.1.6, 7.1.7, 7.1.8, 7.1.9, 7.1.10, 7.2.0, 7.2.1, 7.2.2, 7.2.3, 7.2.4, 7.2.5, 7.2.6, 7.2.7, 7.2.8, 7.2.9, 7.2.10, 7.3.0, 7.3.1, 7.3.2, 7.3.3, 7.3.4, 7.3.5, 7.3.6, 7.3.7, 7.3.8, 7.3.9, 8.0.0, 8.0.1, 8.0.2, 8.0.3, 8.0.4, 8.0.5, 8.0.6, 8.0.7, 8.0.8, 8.0.9, 8.0.10
Feedback submitted, thanks!