Upgrade an indexer cluster

The upgrade process differs considerably depending on the nature of the upgrade. This topic covers these version-based scenarios:

• Upgrading from 7.1 or higher
• Upgrading from 6.x to 7.1
• Upgrading to a new maintenance release (for example, from 8.1.1 to 8.1.2)

In addition, the topic describes:

• How to upgrade an indexer cluster that integrates with a search head cluster.
• How to perform a site-by-site upgrade of a multisite indexer cluster.

Migrating from single-site to multisite?

To convert a single-site indexer cluster to multisite, perform the upgrade first and then read Migrate an indexer cluster from single-site to multisite.

Upgrading an indexer cluster that integrates with a search head cluster?

Upgrade the search head cluster as part of the procedure for performing an indexer cluster rolling upgrade. See Perform a rolling upgrade of a Splunk Enterprise indexer cluster.

If you don't want to perform an indexer cluster rolling upgrade, you can instead follow the steps for tiered upgrades. See Upgrade each tier separately.

Upgrading an indexer cluster that does not have a custom security key?

The security key, also known as the pass4SymmKey setting, authenticates communication between the manager node and the peers and search heads.

Starting in 6.6, a non-default security key is required. If the cluster's security key was never explicitly set to a custom value, a warning message appears on the manager node:

pass4SymmKey setting in the clustering or general stanza of server.conf is set to empty or the default value. You must change it to a different value.

To remediate this situation, you must set the security key on all the cluster nodes (manager node, peer nodes, search heads) while the cluster is down. The key must be the same across all cluster nodes.

You set the security key with the pass4SymmKey attribute in server.conf. See Configure the security key.

To set the key during cluster upgrade, you must upgrade all cluster tiers at once, following the procedure in Upgrade all tiers at once. Set the security key while all nodes are down, so that they all have the same security key when they start up.

Use a rolling upgrade for a 7.1 or higher indexer cluster

When upgrading a 7.1 or higher indexer cluster, the preferred method is to perform a rolling upgrade. The rolling upgrade method lets you upgrade peer nodes to the new version with minimal interruption of ongoing searches. See Perform a rolling upgrade of a Splunk Enterprise indexer cluster.

Upgrade all peer nodes in a single operation

Using this method, you take down all peer nodes at the same time during the upgrade operation. You can upgrade all tiers of the cluster (manager node, search heads, peer nodes) at once or you can upgrade each tier separately.

This method is necessary when upgrading a 6.x cluster. However, it can also be used for the upgrade of a cluster of any version. It has the disadvantage of requiring downtime across the entire peer node tier, affecting indexing and searching during that time,

Note: If you have a multisite cluster, you can, in most cases, instead upgrade one site at a time. See Site-by-site upgrade for multisite indexer clusters.

The approach of upgrading each tier separately is particularly useful if the search head tier consists of a search head cluster, because it eliminates the need to upgrade the search head cluster simultaneously with the indexer cluster peer nodes.

Caution: Even when upgrading each tier separately, it is strongly recommended that you complete the entire upgrade process quickly, to avoid any possibility of incompatibilities between node types running different versions.

To upgrade all cluster tiers at once, see Upgrade all tiers at once.

To upgrade each tier separately, see Upgrade each tier separately.

Upgrade all tiers at once

Perform the following steps:

1. Stop the manager node.
2. Stop all the peers and search heads.
   When bringing down the peers, use the splunk stop command, not splunk offline.
3. If the cluster does not use a non-default (custom) security key, you must set one now. Starting in 6.6, indexer clusters require a non-default security key. This key must be the same across all nodes in the cluster. See Upgrading an indexer cluster that does not have a custom security key?. On each node (manager, peers, and search heads), set the key using the procedure in Configure the security key.
4. Upgrade the manager node, following the normal procedure for any Splunk Enterprise upgrade, as described in How to upgrade Splunk Enterprise in the Installation Manual. Do not upgrade the peers yet.
5. Start the manager node, accepting all prompts, if it is not already running.
6. Run splunk enable maintenance-mode on the manager node. To confirm that the manager node is in maintenance mode, run splunk show maintenance-mode. This step prevents unnecessary bucket fix-ups. See Use maintenance mode.
7. Upgrade the search heads, followed by the peer nodes. Use the normal procedure for any Splunk Enterprise upgrade, as described in How to upgrade Splunk Enterprise in the Installation Manual.
8. Start the peer nodes and search heads, if they are not already running.
9. Run splunk disable maintenance-mode on the manager node. To confirm that the manager node is not in maintenance mode, run splunk show maintenance-mode.

You can view the manager node dashboard to verify that all cluster nodes are up and running.

Upgrade each tier separately

When upgrading tiers separately:

• You must upgrade the tiers in the prescribed order.
• Within each tier, you must upgrade all nodes as a single operation.

Functionality introduced in the new release will not be available until all tiers complete the upgrade.

Caution: Even when upgrading each tier separately, it is strongly recommended that you complete the entire upgrade process quickly, to avoid any possibility of incompatibilities between node types running different versions.

You must follow this order of upgrade when upgrading the tiers in discrete operations:

1. Upgrade the manager node.
2. Upgrade the search head tier.
3. Upgrade the peer node tier.

1. Upgrade the manager node

1. Stop the manager node.
2. Upgrade the manager node, following the normal procedure for any Splunk Enterprise upgrade, as described in How to upgrade Splunk Enterprise in the Installation Manual.
3. Start the manager node, accepting all prompts, if it is not already running.

You can view the manager node dashboard to verify that all cluster nodes are up and running.

2. Upgrade the search head tier

The method that you use to upgrade the search head tier depends on whether or not the tier consists of a search head cluster:

• If the search head tier consists of a search head cluster, follow the procedure in Upgrade a search head cluster. If desired, you can perform a rolling upgrade of the search head cluster, as described in that topic.

• If the search head tier consists of independent search heads, follow this procedure:

1. Stop all the search heads.
2. Upgrade the search heads, following the normal procedure for any Splunk Enterprise upgrade, as described in How to upgrade Splunk Enterprise in the Installation Manual.
3. Start the search heads, if they are not already running.

You can view the manager node dashboard to verify that all cluster nodes are up and running.

3. Upgrade the peer node tier

1. Run splunk enable maintenance-mode on the manager node.
   To confirm that the manager node is in maintenance mode, run splunk show maintenance-mode on the manager node.
   This step prevents unnecessary bucket fix-ups. See Use maintenance mode.
2. Stop all the peer nodes.
   When bringing down the peers, use the splunk stop command, not splunk offline.
3. Upgrade the peer nodes, following the normal procedure for any Splunk Enterprise upgrade, as described in How to upgrade Splunk Enterprise in the Installation Manual.
4. Start the peer nodes, if they are not already running.
5. Run splunk disable maintenance-mode on the manager node.
   To confirm that the manager node is not in maintenance mode, run splunk show maintenance-mode on the manager node.

You can view the manager node dashboard to verify that all cluster nodes are up and running.

Site-by-site upgrade for multisite indexer clusters

Caution: You cannot use this method to upgrade a cluster with metrics data from 7.3.x or earlier to 8.0.x or later, due to a change introduced in 8.0 in how metrics data is stored. Instead, you must either use the rolling upgrade method described in Perform a rolling upgrade of a Splunk Enterprise indexer cluster or take down and upgrade all peer nodes as a single operation, as described in the methods covered by Upgrade all peer nodes in a single operation. As you will note when studying each of these methods, they all put the cluster into maintenance mode for the duration of the peer upgrades. This step is essential, because it ensures that no replication of metrics data takes place between a 7.3.x peer and an 8.0.x peer.

If you have a multisite cluster, you can upgrade one site at a time, as long as you are upgrading across no more than one sequential n.n version (for example, from 6.5 to 6.6, or 6.6 to 7.0, but not 6.5 to 7.0). Because each site has a full set of primary copies, this method allows searches to continue uninterrupted during the upgrade.

You cannot perform a site-by-site upgrade if you are upgrading across more than one sequential n.n version (for example, from 6.4 to 6.6 or 6.5 to 7.0). To upgrade across multiple sequential n.n versions, you must either use the rolling upgrade method described in Perform a rolling upgrade of a Splunk Enterprise indexer cluster or take down and upgrade all peer nodes as a single operation, as described in the methods covered by Upgrade all peer nodes in a single operation.

Alternatively, to upgrade across multiple sequential n.n versions, you can upgrade via interim releases of not more than a single n.n version. For example, if you are upgrading from 6.4 to 6.6, you can first upgrade to a 6.5 interim release using the site-by-site method. You can then upgrade to 6.6.

Functionality introduced in the new release will not be available until all nodes complete the upgrade.

For a two-site cluster, the upgrade procedure has three distinct phases:

1. Upgrade of the manager node.

2. Upgrade of the site1 peers and search heads.

3. Upgrade of the site2 peers and search heads.

Here are the steps in detail:

1. Stop the manager node.

2. Upgrade the manager node, following the normal procedure for any Splunk Enterprise upgrade, as described in How to upgrade Splunk Enterprise in the Installation Manual.

3. Start the manager node, accepting all prompts, if it is not already running.

4. Run splunk enable maintenance-mode on the manager node. To confirm that the manager is in maintenance mode, run splunk show maintenance-mode. This step prevents unnecessary bucket fix-ups. See Use maintenance mode.

5. Stop all the peers and search heads on site1 with the splunk stop command.

6. Upgrade the site1 peer nodes and search heads.

7. Start the site1 peer nodes and search heads, if they are not already running.

8. Run splunk disable maintenance-mode on the manager node. To confirm that the manager is not in maintenance mode, run splunk show maintenance-mode.

9. Wait until the manager node dashboard shows that both the search factor and replication factor are met.

10. Run splunk enable maintenance-mode on the manager node. To confirm that the manager is in maintenance mode, run splunk show maintenance-mode.

11. Stop all the peers and search heads on site2 with the splunk stop command.

12. Upgrade the site2 peer nodes and search heads.

13. Start the site2 peer nodes and search heads, if they are not already running.

14. Run splunk disable maintenance-mode on the manager node. To confirm that the manager is not in maintenance mode, run splunk show maintenance-mode.

You can view the manager node dashboard to verify that all cluster nodes are up and running.

Upgrade to a maintenance release

To upgrade a cluster to a maintenance release (for example, from 8.1.1 to 8.1.2), you do not need to bring down the entire cluster at once. Instead, you can perform a rolling, online upgrade, in which you upgrade the nodes one at a time.

Caution: Even with a rolling upgrade, you should upgrade all nodes quickly, for several reasons:

• Proper functioning of the cluster depends on all peer nodes running the same version of Splunk Enterprise, as stated in System requirements and other deployment considerations for indexer clusters.
• Other version compatibility requirements must also be met, as described in System requirements and other deployment considerations for indexer clusters.
• If you upgrade the manager node but not the peers, the cluster might generate errors and warnings. This is generally okay for a short duration, but you should complete the upgrade of all nodes as quickly as possible.

To upgrade a cluster node, follow the normal procedure for any Splunk Enterprise upgrade, with the few exceptions described below. For general information on upgrading Splunk Enterprise instances, see How to upgrade Splunk Enterprise.

To perform a rolling maintenance upgrade, follow these steps:

1. Upgrade the manager node

Upgrade the manager node first.

For information on what happens when the manager node goes down, as well as what happens when it comes back up, see What happens when the manager node goes down.

2. Upgrade the search heads

The only impact to the cluster when you upgrade the search heads is disruption to searches during that time.

3. Put the manager node into maintenance mode

Run splunk enable maintenance-mode on the manager node. To confirm that the manager is in maintenance mode, run splunk show maintenance-mode. This step prevents unnecessary bucket fix-ups. See Use maintenance mode.

4. Upgrade the peer nodes

When upgrading peer nodes, note the following:

• Peer upgrades can disrupt ongoing searches.

• To minimize downtime and to limit any disruption to indexing and searching, upgrade the peer nodes one at a time.

• To bring a peer down prior to upgrade, use the splunk offline command, as described in Take a peer offline.

• During the interim between when you upgrade the manager node and when you finish upgrading the peers, the cluster might generate various warnings and errors.

• For multisite clusters, the site order of peer upgrades does not matter.

5. Take the manager node out of maintenance mode

Run splunk disable maintenance-mode on the manager node. To confirm that the manager is not in maintenance mode, run splunk show maintenance-mode.

Upgrade a cluster that uses cluster manager redundancy

When upgrading a cluster with cluster manager redundancy, the best practice to ensure continued high availability of a manager throughout the process is to upgrade the standby managers first, followed by the active manager:

1. Put all cluster managers into maintenance mode by running splunk enable maintenance-mode on each manager. To confirm that each manager is in maintenance mode, run splunk show maintenance-mode on each manager.
2. For each standby manager:
1. Stop the manager.
2. Upgrade the manager, following the normal procedure for any Splunk Enterprise upgrade, as described in How to upgrade Splunk Enterprise in the Installation Manual.
3. Start the manager.
4. Check that the manager enters standby mode and resyncs with the active manager by running splunk cluster-manager-redundancy -show-status.
3. Upgrade the active manager:
   1. Switch one of the standby managers to active mode by running splunk cluster-manager-redundancy -switch-mode active on the designated new active manager. Check that the switch was successful by running splunk cluster-manager-redundancy -show-status.
   2. Stop the previously active manager.
   3. Upgrade the manager.
   4. Start the manager.
   5. Check that the manager enters standby mode and resyncs with the new active manager.
4. Upgrade the search heads and peer nodes in the usual fashion.
5. (Optional) Switch the original active manager back to active mode.
6. Take all managers out of maintenance mode by running splunk disable maintenance-mode on each manager.