Splunk® Enterprise

Distributed Search

Download manual as PDF

This documentation does not apply to the most recent version of Splunk. Click here for the latest version.
Download topic as PDF

Restart the search head cluster

You can restart the entire cluster with the splunk rolling-restart command. The command performs a phased restart of all cluster members, so that the cluster as a whole can continue to perform its functions during the restart process.

The deployer also automatically initiates a rolling restart, when necessary, after distributing a configuration bundle to the members. For details on this process, see "Push the configuration bundle".

Caution: In most cases, when changing configuration settings in the [shclustering] stanza of server.conf, you must restart all members at approximately the same time, in order to maintain identical settings across all members. For this reason, do not use the splunk rolling-restart command to restart the members after such configuration changes, except when configuring the captain_is_adhoc_searchhead attribute. Instead, run the splunk restart command on each member. See "Configure the search head cluster".

Initiate a rolling restart

Invoke the splunk rolling-restart command from the captain:

splunk rolling-restart shcluster-members

How rolling restart works

The rolling restart works like this: The captain issues a restart message to approximately 10%, by default, of the members at a time. Once those members restart and contact the captain, the captain then issues a restart message to another 10% of the members, and so on, until all the members, including the captain, have restarted.

Note: If there are fewer than 10 members in the cluster, the captain issues the restart to one member at a time.

The captain is the final member to restart. Restart of the captain triggers the election process, which can result in a new captain.

After all members have restarted, it requires approximately 60 seconds for the cluster to stabilize. During this interval, error messages might appear. You can ignore these messages. They should desist within 60 seconds.

Note: During a rolling restart, there is no guarantee that all knowledge objects will be available to all members.

Configure the number of members that restart simultaneously

By default, the captain issues the restart command to 10% of the members at a time. However, the percentage is configurable through the percent_peers_to_restart attribute in the [shclustering] stanza of server.conf. For convenience, you can configure this attribute with the CLI splunk edit shcluster-config command. For example, to change the restart behavior so that the captain restarts 20% of the peers at a time, use this command:

splunk edit shcluster-config -percent_peers_to_restart 20

Caution: Do not set the value to greater than 20%. Otherwise, issues can arise during the captain election process.

After changing the percent_peers_to_restart attribute, you still need to run the splunk rolling-restart command to initiate the actual restart.

Monitor the restart process

To check the progress of the rolling restart, run this command from any of the members:

splunk show shcluster-status -auth <username>:<password>

This command returns, among other values, a rolling_restart_flag that indicates whether a rolling-restart is in progress (1) or not (0).

Note: This command is not operative during the final step of the restart process, when the captain itself, which tracks this information, is restarting. During that time, you might see an error message that begins, "In handler 'shclusterstatus': Node is not captain...." Wait approximately 60 seconds for the restart process to complete and retry the command.

PREVIOUS
Handle failure of a search head cluster member
  NEXT
Use the CLI to view information about a search head cluster

This documentation applies to the following versions of Splunk® Enterprise: 6.2.0, 6.2.1, 6.2.2, 6.2.3, 6.2.4, 6.2.5, 6.2.6, 6.2.7, 6.2.8, 6.2.9, 6.2.10, 6.2.11, 6.2.12, 6.2.13, 6.2.14, 6.2.15


Comments

Ben - I don't know of any curl command for initiating a rolling restart. The set of available REST endpoints is documented here: http://docs.splunk.com/Documentation/Splunk/latest/RESTREF/RESTcluster

At this point, the CLI 'splunk rolling-restart' command requires that you run it from the captain.

Sgoodman, Splunker
July 6, 2015

Can Splunk provide the curl command to initiate a rolling restart? The use case is to run the command and initiate a rolling restart if we do not know which node is the captain. By hitting all nodes of a cluster, the rolling restart is bound to happen.

Ben leung
June 29, 2015

Was this documentation topic helpful?

Enter your email address, and someone from the documentation team will respond to you:

Please provide your comments here. Ask a question or make a suggestion.

You must be logged into splunk.com in order to post comments. Log in now.

Please try to keep this discussion focused on the content covered in this documentation topic. If you have a more general question about Splunk functionality or are experiencing a difficulty with Splunk, consider posting a question to Splunkbase Answers.

0 out of 1000 Characters