Control captaincy

You have considerable control over which members become captain, through these methods:

• You can designate members as either "preferred captains" or "not preferred captains." When the cluster assigns captaincy, it attempts to assign it to a member with a preferred captain designation.
• You can transfer captaincy from one member to another.

In addition, by default, the cluster attempts to prevent an out-of-sync member from becoming captain. An out-of-sync member is one whose set of replicated configurations is out of sync with that of the current or most recent captain.

See Search head cluster captain for details on the captain's role in a search head cluster.

Use cases

It can be useful to control captaincy to handle a number of situations. For example:

• You have one member that you want to always use as captain. Or conversely, you have one member that you never want to be captain.
• You do not want the captain to perform any user-initiated ad hoc jobs. You can achieve this by designating one specific member as captain and keeping your third-party load balancer ignorant of that member.
• You want to repair the state of the cluster. A quick way to do this is to switch to a new captain, because members join a new captain in a clean state.

The twin tools of preferred captaincy and captaincy transfer give you flexibility when you need to control captaincy. Although neither one can guarantee that you always maintain complete control over the location of your captain, they do limit the likelihood that the captain will reside on a member that is not optimal for your needs. And captaincy transfer offers the ability to transfer the captain to a new member as needed.

Specify captaincy preference

You can designate some members as preferred captains and others as non-preferred captains. When the cluster assigns captaincy through the election process, it attempts to assign it to a member with a preferred captain designation.

Designate captaincy preference

To specify a member's preference for captaincy, set the preferred_captain attribute in that member's server.conf file:

preferred_captain = true|false

This attribute defaults to true, which means that, by default, all members are preferred captains.

To limit the likelihood that the cluster will assign captaincy to a particular member, set that member's preferred_captain attribute to false:

preferred_captain = false

The cluster attempts to respect the captaincy preference.

Limitations of captaincy preference

The cluster tries to assign captaincy to a member with preferred_captain=true. However, it might not always be possible to assign captaincy to a preferred-captain member. For example, if none of the preferred-captain members are reachable over the network, then captaincy might be assigned to a member with preferred_captain=false.

During an election for a new captain, a non-preferred-captain member can briefly become the captain before captaincy transfers to a preferred-captain member. If no preferred-captain members are available, the non-preferred-captain member remains captain until a preferred-captain member becomes available.

Prevent out-of-sync members from becoming captain

By default, the cluster attempts to prevent an out-of-sync member from becoming captain.

What is an out-of-sync member?

An out-of-sync member is a member that cannot sync its own set of replicated configurations with the common baseline set of replicated configurations maintained by the current or most recent captain. You do not want an out-of-sync member to become captain.

The captain maintains the baseline set of configurations for all members. When a configuration change occurs on one member, the member sends the change to the captain, which then replicates the change to all the other members. Therefore, it is essential that the baseline set of configurations on the captain be up-to-date.

If a member's set of configurations differs from the captain's baseline set, the member is considered to be out-of-sync. This can occur, for example, if the member lost network connectivity with the cluster for an extended period of time. When the member returns to the cluster, it needs to resync with the baseline set of configurations. If a large number of configuration changes occurred while the member was not in contact with the cluster, the resync can require manual intervention.

While a member is out-of-sync, it must not become captain. If an out-of-sync member does become captain, its set of configurations becomes the baseline that gets replicated to all other members. This situation would result in the loss of configuration changes made on other members.

See Replication synchronization issues.

Set out-of-sync behavior

The prevent_out_of_sync_captain attribute in server.conf determines whether the cluster considers out-of-sync status when evaluating a member's eligibility for captain.

By default, this attribute is set to true. That is, the cluster attempts to prevent the member from becoming captain if it is out-of-sync. It is extremely unlikely that you will need to change this default behavior.

This attribute must be set to the same value on all members.

How the cluster determines member eligibility for captain

When electing a captain, the cluster considers the out-of-sync state to be more important the preferred-captain state. That is, if all preferred-captain members are out-of -sync, the cluster attempts to elect as captain a non-preferred-captain member, rather than a preferred-captain member that is out-of-sync. Briefly, here is the order that the cluster uses to determine member eligibility for captain:

1. Preferred-captain members that are not out-of-sync
2. Non-preferred-captain members that are not out-of-sync
3. A preferred-captain member that is out-of-sync
4. A non-preferred-captain member that is out-of-sync

This order assumes that you maintain the default behavior of prevent_out_of_sync_captain=true.

Transfer captaincy

You can transfer captaincy from one member to another.

The use of captaincy transfer does not interfere with the normal captain election process, which always proceeds in response to the circumstances described in Captain election. If an election occurs and results in the captain moving to a member other than the one you want it to reside on, you can then invoke captaincy transfer to relocate the captain.

Change the captain

To transfer captaincy to a different member, run this command from any member:

splunk transfer shcluster-captain -mgmt_uri <URI>:<management_port> -auth <username>:<password>

Note the following:

• The -mgmt_uri parameter specifies the URI and management port for the member that you want to transfer captaincy to. You must use the fully qualified domain name.
• You can run this command from any member. You are not limited to running it from the current captain or the intended captain.
• You do not need to restart any member after running the command.

To confirm that the captaincy transfer was successful, run the splunk show shcluster-status command from any member:

splunk show shcluster-status -auth <username>:<password>

Among other information returned, this command identifies the current captain.

You can also transfer captaincy through the search head clustering dashboard in Settings. See Use the search head clustering dashboard.

Some ways to employ captaincy transfer in scripts

The splunk transfer shcluster-captain command can be useful for scripting certain cluster behavior. For example:

• To ensure that captaincy stays with a particular member, you can implement a cron job that monitors the captain on a periodic basis. If the check detects a change in captain, it can automatically run the splunk transfer shcluster-captain command to return captaincy to the preferred member.
• To implement rolling-restart-style functionality (for example, if deploying cluster updates through some third-party tool), you can transfer captaincy to another member prior to restarting the current captain.

Captaincy transfer and rolling-restarts

As of 6.3, the rolling-restart process automatically invokes captaincy transfer to prevent captaincy from changing during the restart process. Because of this action, the member that was captain prior to the restart ordinarily continues as captain after the restart. See Restart the search head cluster.

Captaincy transfer and static captain

Captaincy transfer is available only with a dynamic captain. For information on the use of a static captain for disaster recovery, see Use static captain to recover from loss of majority.