Use rolling upgrade
Splunk Enterprise version 7.1 and later supports rolling upgrades for search head clusters. A rolling upgrade performs a phased upgrade of cluster members with minimal interruption of ongoing searches. You can use rolling upgrade to minimize search disruption, when upgrading cluster members to a new version of Splunk Enterprise.
Requirements and considerations
- Rolling upgrade only applies to upgrades from version 7.1 to later versions of Splunk Enterprise.
- All search head cluster members, cluster master, and all peer nodes must be running version 7.1 or later.
- Do not attempt any clustering maintenance operations, such as rolling restart, bundle pushes, or node additions, during upgrade.
In case of hardware or network failures that prevent node shutdown or restart, manual intervention might be required.
How a rolling upgrade works
When you initiate a rolling upgrade, you select a cluster member and put that member into manual detention. While in manual detention, the member cannot accept new search jobs, and all in-progress searches try to complete within a configurable timeout. When all searches are complete, you perform the software upgrade, and bring the member back online. You repeat this process for each cluster member until the rolling upgrade is complete.
Things to note about the behavior of a rolling upgrade:
- Cluster members are upgraded one at a time.
- While in manual detention, a member:
- cannot receive new searches (new scheduled searches are executed on other members).
- cannot execute ad hoc searches.
- cannot receive new search artifacts from other members.
- continues to participate in cluster operations.
- The member waits for in-progress searches to complete, up to a maximum time set by the user. The default of 180 seconds is enough time for the majority of searches to complete in most cases.
- Rolling upgrades apply to both historical and real-time searches.
Perform a rolling upgrade
To perform a rolling upgrade:
- Run preliminary health checks.
- Initialize the rolling upgrade.
- Put a cluster member into manual detention.
- Confirm the member is ready for upgrade by checking the status of existing search jobs.
- Upgrade the member.
- Bring the member back online.
- Start the member.
- Turn off manual detention mode on the upgraded member.
- Check the cluster health status before continuing with the rolling upgrade.
- Repeat steps 3-7 for all members.
- Upgrade the deployer.
- Finalize the rolling upgrade.
1. Run preliminary health checks
On any cluster member, run the splunk show shcluster-status
command using the verbose
option to confirm that the cluster is in a healthy state before you begin the upgrade.
splunk show shcluster-status --verbose
Here is an example of the output from the above command:
Captain: decommission_search_jobs_wait_secs : 180 dynamic_captain : 1 elected_captain : Tue Mar 6 23:35:52 2018 id : FEC6F789-8C30-4174-BF28-674CE4E4FAE2 initialized_flag : 1 label : sh3 max_failures_to_keep_majority : 1 mgmt_uri : https://sroback180306192122accme_sh3_1:8089 min_peers_joined_flag : 1 rolling_restart : restart rolling_restart_flag : 0 rolling_upgrade_flag : 0 service_ready_flag : 1 stable_captain : 1 Cluster Master(s): https://sroback180306192122accme_master1_1:8089 splunk_version: 7.1.0 Members: sh3 label : sh3 manual_detention : off mgmt_uri : https://sroback180306192122accme_sh3_1:8089 mgmt_uri_alias : https://10.0.181.9:8089 out_of_sync_node : 0 preferred_captain : 1 restart_required : 0 splunk_version : 7.1.0 status : Up sh2 label : sh2 last_conf_replication : Wed Mar 7 05:30:09 2018 manual_detention : off mgmt_uri : https://sroback180306192122accme_sh2_1:8089 mgmt_uri_alias : https://10.0.181.4:8089 out_of_sync_node : 0 preferred_captain : 1 restart_required : 0 splunk_version : 7.1.0 status : Up sh1 label : sh1 last_conf_replication : Wed Mar 7 05:30:09 2018 manual_detention : off mgmt_uri : https://sroback180306192122accme_sh1_1:8089 mgmt_uri_alias : https://10.0.181.2:8089 out_of_sync_node : 0 preferred_captain : 1 restart_required : 0 splunk_version : 7.1.0 status : Up
The output shows a stable, dynamically elected captain, enough members to support the replication factor, no out-of-sync nodes, and all members running a compatible Splunk Enterprise version (7.1.0 or later). This indicates the cluster is in a healthy state to perform a rolling upgrade.
For information on health check criteria, see Health check output details.
Health checks are not all inclusive. Checks apply only to the criteria listed.
Or, use this endpoint to monitor cluster health:
/services/shcluster/status?advanced=1
For endpoint details, see shcluster/status in the REST API Reference Manual.
Based on health check results, either fix any issues impacting cluster health, or proceed with caution and continue the upgrade.
2. Initialize rolling upgrade
To start the rolling upgrade process, on any member, run the following CLI command:
splunk upgrade-init shcluster-members
Or, send a POST request to:
/services/shcluster/captain/control/control/upgrade-init
For endpoint details, see shcluster/captain/control/control/upgrade-init in the REST API Reference Manual.
3. Put a member into manual detention mode
Select a search head cluster member (other than the captain) and put that member into manual detention mode:
splunk edit shcluster-config -manual_detention on
Or, send a POST request to:
servicesNS/admin/search/shcluster/member/control/control/set_manual_detention \ -d manual_detention=on
For endpoint details, see shcluster/member/control/control/set_manual_detention in the REST API Reference Manual.
For more information on manual detention mode, see Put a search head into detention.
4. Confirm the member is ready for upgrade
Before you proceed with the upgrade, run the following command to confirm that all searches are complete:
splunk list shcluster-member-info | grep "active"
The following output indicates that all historical and realtime searches are complete:
active_historical_search_count:0 active_realtime_search_count:0
Or send a GET request against:
/services/shcluster/member/info
For endpoint details, see shcluster/member/info in the REST API Reference Manual.
5. Upgrade the member
Upgrade the search head, following the normal procedure for any Splunk Enterprise upgrade, as described in How to upgrade Splunk Enterprise in the Installation Manual.
6. Bring the member back online
- On the member, run following command:
splunk start
On restart, the first member upgraded is automatically elected as cluster captain. This captaincy transfer occurs only once during the rolling upgrade.
- Turn off manual detention mode:
splunk edit shcluster-config -manual_detention off
Or, send a POST request to:
servicesNS/admin/search/shcluster/member/control/control/set_manual_detention \ -d manual_detention=off
For endpoint details, see shcluster/member/control/control/set_manual_detention in the REST API Reference Manual.
7. Check cluster health status
After you bring the member back online, check that the cluster is in a healthy state:
splunk show shcluster-status --verbose
Or, use this endpoint to monitor cluster health:
/services/shcluster/status?advanced=1
For endpoint details, see shcluster/status in the REST API Reference Manual.
For information on what determines a healthy search head cluster, see Health check output details.
8. Repeat steps 3-7 for all members
Repeat steps 3-7 above until upgrade of all members is complete.
9. Upgrade the deployer
It is strongly advised that you upgrade the deployer at the same time that you upgrade cluster members. The deployer must run the same version as cluster members, down to the minor level. For example, if members are running 7.1.1, the deployer must run 7.1.x.
- Stop the deployer.
- Upgrade the deployer, following the normal procedure for any Splunk Enterprise upgrade, as described in How to upgrade Splunk Enterprise in the Installation Manual.
- Start the deployer.
For more information on the deployer, see Deployer requirements.
10. Finalize the rolling upgrade
On any SHC member, run the following CLI command:
splunk upgrade-finalize shcluster-members
Or, send a POST request to:
/services/shcluster/captain/control/control/upgrade-finalize
For endpoint details, see shcluster/captain/control/control/upgrade-finalize in the REST API Reference Manual.
Example upgrade automation script
Version 7.1 includes an example automation script (shc_upgrade_template.py
) that you can use as the basis for automating the search head cluster rolling upgrade process. This is a template script that should not be used as is, but should be modified based on your deployment.
shc_upgrade_template.py
is located in SPLUNK_HOME/bin
and includes detailed usage and workflow information.
shc_upgrade_template.py
is an example script only. It should not be applied to a production instance without editing to suit your environment and testing extensively.
Upgrade a search head cluster | Configure the search head cluster |
This documentation applies to the following versions of Splunk® Enterprise: 7.1.0, 7.1.1, 7.1.2, 7.1.3, 7.1.4, 7.1.5, 7.1.6, 7.1.7, 7.1.8, 7.1.9, 7.1.10
Feedback submitted, thanks!