Upgrade a search head cluster

This topic describes how to upgrade a search head cluster. The process is the same for maintenance and major release upgrades.

Starting with version 6.5, you can perform a member-by-member upgrade. This lets you perform a phased upgrade of cluster members that allows the cluster to continue operating during the upgrade. To use the member-by-member upgrade process, you must be upgrading from version 6.4 or higher.

Starting with version 7.1, you can perform a rolling upgrade. Rolling upgrade lets you perform a phased upgrade of cluster members with minimal interruption of ongoing searches. To use rolling upgrade, you must be upgrading from version 7.1 or higher. For more information, see Perform a rolling upgrade of a search head cluster.

Starting with version 9.1, you can perform an automated rolling upgrade of a search head cluster. Automated rolling upgrade incorporates existing rolling upgrade functionality to minimize the number of steps required to upgrade search head cluster members to a new version of Splunk Enterprise. To perform an automated rolling upgrade, you must be upgrading from version 9.1 or higher. For more information, see Perform an automated rolling upgrade of a search head cluster.

Perform an offline upgrade

In a regular offline upgrade, all cluster members are down for the duration of the upgrade process.

You must perform an offline upgrade when upgrading from version 6.3 or earlier.

Before performing the offline upgrade, note the following requirements:

• All cluster members must run the same version of Splunk Enterprise (down to the maintenance level).
• You can run search head cluster members against 5.x or later non-clustered search peers, so it is not necessary to upgrade standalone indexers at the same time. See Splunk Enterprise version compatibility.

Steps

1. Stop all cluster members.
2. Upgrade all members.
3. Stop the deployer.
4. Upgrade the deployer.
5. Start the deployer.
6. Start the members.
7. Wait one to two minutes for captain election to complete. The cluster will then begin functioning.

Perform a member-by-member upgrade

When upgrading from version 6.4 or later, you can perform a member-by-member upgrade.

For a search head cluster that integrates with an indexer cluster, perform a member-by-member upgrade as part of the tiered upgrade procedure. See Upgrade each tier separately in Managing Indexers and Clusters of Indexers.

Before performing the upgrade, note the following requirements:

• Mixed-version clusters are not supported during the ongoing functioning of the cluster. Therefore, you must move quickly through the member-by-member upgrade process, first upgrading one member and then immediately upgrading the next, and so on, until you finish upgrading all members.
• Do not attempt any clustering maintenance operations, such as rolling restart, during upgrade.
• At the end of the upgrade, all members must be running the same version of Splunk Enterprise (down to the maintenance level).
• You can run search head cluster members against 5.x or later non-clustered search peers, so it is not necessary to upgrade standalone indexers at the same time. See Splunk Enterprise version compatibility.

During member-by-member upgrade, KV store replication cannot be guaranteed. For this reason, there must be no KV store activity during the upgrade. To ensure there is no KV store activity during upgrade, perform an offline upgrade instead.

To perform a member-by-member upgrade:

1. Upgrade one member and make it captain:
   1. Stop the member.
   2. Upgrade the member.
   3. Start the member and wait while it joins the cluster.
   4. Transfer captaincy to the upgraded member. See Transfer captaincy.
2. For each additional member, one-by-one:
   1. Stop the member.
   2. Upgrade the member.
   3. Start the member.
3. Upgrade the deployer:
   1. Stop the deployer.
   2. Upgrade the deployer.
   3. Start the deployer.

When upgrading from a version prior to 7.1.0 to version 7.1.0 or higher, you must transfer the captaincy to the first upgraded member. If for any reason the upgraded captain goes down before the cluster upgrade is complete, and a non-upgraded cluster member is elected captain, you must manually transfer the captaincy back to the upgraded member to resolve the problem.

Perform a rolling upgrade

For instructions on how to perform a rolling upgrade with minimal search disruption, see Perform a rolling upgrade of a search head cluster.

For instructions on how to perform an automated rolling upgrade, see Perform an automated rolling upgrade of a search head cluster.

Deployer initiates restart after post-6.2.6 upgrade

The deployer handles user configurations differently in versions higher than 6.2.6, compared to versions 6.2.6 and below. Because of this change, the first time that you use the deployer to distribute updates after upgrading your cluster to a version higher than 6.2.6, the deployer must initiate a rolling restart of all cluster members.

This restart takes place the first time, post-upgrade, that you run the splunk apply shcluster-bundle command. The restart only occurs if you had used the deployer to push user configurations in 6.2.6 or below.

This change in user configuration deployment means that such configurations no longer reside in default directories on the cluster members. This enables certain runtime operations on the configurations. Specifically, you can now delete or move the configurations or change their sharing levels. For more information on how the deployer handles user configurations post-6.2.6, see User configurations.

Changed behavior in 6.5 for user-based and role-based search quotas

The default behavior for handling user-based and role-based concurrent search quotas has changed with version 6.5.

In versions 6.3 and 6.4, the default is to enforce the quotas across the set of cluster members. Starting with 6.5, the default is to enforce the quotas on a member-by-member basis.

You can change quota enforcement behavior, if necessary. See Job scheduling.