Use rolling upgrade

Splunk Enterprise version 7.1 and later supports rolling upgrades for search head clusters. A rolling upgrade performs a phased upgrade of cluster members with minimal interruption of ongoing searches. You can use rolling upgrade to minimize search disruption, when upgrading cluster members to a new version of Splunk Enterprise.

Requirements and considerations

• Rolling upgrade only applies to upgrades from version 7.1 to later versions of Splunk Enterprise.
• All search head cluster members, cluster master, and all peer nodes must be running version 7.1 or later.
• Do not attempt any clustering maintenance operations, such as rolling restart, bundle pushes, or node additions, during upgrade.

In case of hardware or network failures that prevent node shutdown or restart, manual intervention might be required.

How a rolling upgrade works

When you initiate a rolling upgrade, you select a cluster member and put that member into manual detention. While in manual detention, the member cannot accept new search jobs, and all in-progress searches try to complete within a configurable timeout. When all searches are complete, you perform the software upgrade, and bring the member back online. You repeat this process for each cluster member until the rolling upgrade is complete.

Things to note about the behavior of a rolling upgrade:

• Cluster members are upgraded one at a time.
• While in manual detention, a member:
  • cannot receive new searches (new scheduled searches are executed on other members).
  • cannot execute ad hoc searches.
  • cannot receive new search artifacts from other members.
  • continues to participate in cluster operations.
• The member waits for in-progress searches to complete, up to a maximum time set by the user. The default of 180 seconds is enough time for the majority of searches to complete in most cases.
• Rolling upgrades apply to both historical and real-time searches.

Perform a rolling upgrade

To perform a rolling upgrade:

1. Run preliminary health checks.
2. Initialize the rolling upgrade.
3. Put a cluster member into manual detention.
4. Confirm the member is ready for upgrade by checking the status of existing search jobs.
5. Upgrade the member.
6. Bring the member back online.
   1. Start the member.
   2. Turn off manual detention mode on the upgraded member.
7. Check the cluster health status before continuing with the rolling upgrade.
8. Repeat steps 3-7 for all members.
9. Upgrade the deployer.
10. Finalize the rolling upgrade.

1. Run preliminary health checks

On any cluster member, run the splunk show shcluster-status command using the verbose option to confirm that the cluster is in a healthy state before you begin the upgrade.

splunk show shcluster-status --verbose

Here is an example of the output from the above command:

Captain:
		decommission_search_jobs_wait_secs : 180
		               dynamic_captain : 1
		               elected_captain : Tue Mar  6 23:35:52 2018
		                            id : FEC6F789-8C30-4174-BF28-674CE4E4FAE2
		              initialized_flag : 1
		                         label : sh3
		 max_failures_to_keep_majority : 1
		                      mgmt_uri : https://sroback180306192122accme_sh3_1:8089
		         min_peers_joined_flag : 1
		               rolling_restart : restart
		          rolling_restart_flag : 0
		          rolling_upgrade_flag : 0
		            service_ready_flag : 1
		                stable_captain : 1

 Cluster Master(s):
	https://sroback180306192122accme_master1_1:8089		splunk_version: 7.1.0

 Members:
	sh3
		                         label : sh3
		              manual_detention : off
		                      mgmt_uri : https://sroback180306192122accme_sh3_1:8089
		                mgmt_uri_alias : https://10.0.181.9:8089
		              out_of_sync_node : 0
		             preferred_captain : 1
		              restart_required : 0
		                splunk_version : 7.1.0
		                        status : Up
	sh2
		                         label : sh2
		         last_conf_replication : Wed Mar  7 05:30:09 2018
		              manual_detention : off
		                      mgmt_uri : https://sroback180306192122accme_sh2_1:8089
		                mgmt_uri_alias : https://10.0.181.4:8089
		              out_of_sync_node : 0
		             preferred_captain : 1
		              restart_required : 0
		                splunk_version : 7.1.0
		                        status : Up
	sh1
		                         label : sh1
		         last_conf_replication : Wed Mar  7 05:30:09 2018
		              manual_detention : off
		                      mgmt_uri : https://sroback180306192122accme_sh1_1:8089
		                mgmt_uri_alias : https://10.0.181.2:8089
		              out_of_sync_node : 0
		             preferred_captain : 1
		              restart_required : 0
		                splunk_version : 7.1.0
		                        status : Up

The output shows a stable, dynamically elected captain, enough members to support the replication factor, no out-of-sync nodes, and all members running a compatible Splunk Enterprise version (7.1.0 or later). This indicates the cluster is in a healthy state to perform a rolling upgrade.

For information on health check criteria, see Health check output details.

Health checks are not all inclusive. Checks apply only to the criteria listed.

Or, use this endpoint to monitor cluster health:

/services/shcluster/status?advanced=1

For endpoint details, see shcluster/status in the REST API Reference Manual.

Based on health check results, either fix any issues impacting cluster health, or proceed with caution and continue the upgrade.

2. Initialize rolling upgrade

To start the rolling upgrade process, on any member, run the following CLI command:

splunk upgrade-init shcluster-members

Or, send a POST request to:

/services/shcluster/captain/control/control/upgrade-init

For endpoint details, see shcluster/captain/control/control/upgrade-init in the REST API Reference Manual.

3. Put a member into manual detention mode

Select a search head cluster member (other than the captain) and put that member into manual detention mode:

splunk edit shcluster-config -manual_detention on

Or, send a POST request to:

servicesNS/admin/search/shcluster/member/control/control/set_manual_detention \
-d manual_detention=on

For endpoint details, see shcluster/member/control/control/set_manual_detention in the REST API Reference Manual.

For more information on manual detention mode, see Put a search head into detention.

4. Confirm the member is ready for upgrade

Before you proceed with the upgrade, run the following command to confirm that all searches are complete:

splunk list shcluster-member-info | grep "active"

The following output indicates that all historical and realtime searches are complete:

active_historical_search_count:0
active_realtime_search_count:0

Or send a GET request against:

/services/shcluster/member/info

For endpoint details, see shcluster/member/info in the REST API Reference Manual.

5. Upgrade the member

Upgrade the search head, following the normal procedure for any Splunk Enterprise upgrade, as described in How to upgrade Splunk Enterprise in the Installation Manual.

6. Bring the member back online

1. On the member, run following command:

   splunk start
   

   On restart, the first member upgraded is automatically elected as cluster captain. This captaincy transfer occurs only once during the rolling upgrade.

2. Turn off manual detention mode:

   splunk edit shcluster-config -manual_detention off
   

   Or, send a POST request to:

   servicesNS/admin/search/shcluster/member/control/control/set_manual_detention \
   -d manual_detention=off
   

   For endpoint details, see shcluster/member/control/control/set_manual_detention in the REST API Reference Manual.

7. Check cluster health status

After you bring the member back online, check that the cluster is in a healthy state:

splunk show shcluster-status --verbose

Or, use this endpoint to monitor cluster health:

/services/shcluster/status?advanced=1

For endpoint details, see shcluster/status in the REST API Reference Manual.

For information on what determines a healthy search head cluster, see Health check output details.

8. Repeat steps 3-7 for all members

Repeat steps 3-7 above until upgrade of all members is complete.

9. Upgrade the deployer

It is strongly advised that you upgrade the deployer at the same time that you upgrade cluster members. The deployer must run the same version as cluster members, down to the minor level. For example, if members are running 7.1.1, the deployer must run 7.1.x.

1. Stop the deployer.
2. Upgrade the deployer, following the normal procedure for any Splunk Enterprise upgrade, as described in How to upgrade Splunk Enterprise in the Installation Manual.
3. Start the deployer.

For more information on the deployer, see Deployer requirements.

10. Finalize the rolling upgrade

On any SHC member, run the following CLI command:

splunk upgrade-finalize shcluster-members

Or, send a POST request to:

/services/shcluster/captain/control/control/upgrade-finalize

For endpoint details, see shcluster/captain/control/control/upgrade-finalize in the REST API Reference Manual.

Example upgrade automation script

Version 7.1 includes an example automation script (shc_upgrade_template.py) that you can use as the basis for automating the search head cluster rolling upgrade process. This is a template script that should not be used as is, but should be modified based on your deployment.

shc_upgrade_template.py is located in SPLUNK_HOME/bin and includes detailed usage and workflow information.

shc_upgrade_template.py is an example script only. It should not be applied to a production instance without editing to suit your environment and testing extensively.