Put a search head cluster member into detention

You can put a search head cluster member into manual detention to allow for activities such as search head cluster rolling upgrades, rolling restart, or maintenance operations. When a search head cluster member is in manual detention, it stops accepting all new searches from the search scheduler or from users. Existing ad-hoc and scheduled search jobs run to completion. New scheduled searches are distributed by the captain to search head cluster members that are up and not in detention. You can run new ad-hoc searches against other members of the search head cluster. The search head in detention continues to participate in most cluster operations, such as captain election and conf replication, with the exception of search artifact replication.

You can put a search head cluster member in detention via the CLI, REST endpoint, or via the server.conf file.

When you manually put a search head cluster member into the detention state, it remains in detention until you remove it from detention, and the detention state persists through a restart.

This capability is limited to members in a search head cluster. It is not available to stand-alone search heads.

Use cases

Manual detention is useful for cases where you need a search head to be a functional member of a cluster, but you need to perform maintenance of some kind on the search head:

• Rolling upgrades. You can put a search head cluster member in detention as a part of a rolling upgrade. A rolling upgrade is a phased upgrade of all cluster members, so that searches can run without disruption during the upgrade process.
• Search head cluster maintenance. You can put a search head cluster member in detention to perform maintenance. Once the search head cluster member is in detention and all in-progress searches are completed, the member can be removed from the search head cluster for maintenance operations like hardware replacement or OS upgrade.
• Search head diagnostics. You can use manual detention to prevent searches from being sent to a poorly performing search head while you run diagnostics.
• Searchable rolling restarts. Manual detention is used by default in searchable rolling restarts. No action is required.

For information on searchable rolling restarts, see Restart the search head cluster. For information on rolling upgrades, see Use rolling upgrade.

How existing searches are handled

If a search is running on search head cluster member when it is placed in detention, the following behavior occurs:

• On a search head that is in manual detention but not a part of a searchable rolling restart. These searches will run to completion.
• On a search head that is a part of a searchable rolling restart. By default, these searches run for 180 seconds. Or, you can set a timeout period using the decommission_search_jobs_wait_secs attribute in the [shclustering] stanza of the search head's server.conf file. This attribute determines the amount of time, in seconds, that a cluster member waits for existing searches to complete before restarting.
• On a search head that is a part of a rolling upgrade. During rolling upgrade of a search head cluster, you can put a single search head into manual detention and wait for the existing search jobs to run to completion before you shut down the search head.

You can run the following CLI command to confirm that all searches are complete:

splunk list shcluster-member-info | grep "active"

The following output indicates that all historical and realtime searches are complete:

active_historical_search_count:0
active_realtime_search_count:0

Or send a GET request against:

/services/shcluster/member/info

See the documentation for editing the decommission_search_jobs_wait_secs attribute in the server.conf files here: search head clustering configuration.
See the documentation for searchable rolling restarts here: How searchable rolling restart works.

Put a search head cluster member into detention via the CLI

To put a search head cluster member into detention, run the CLI command splunk edit shcluster-config with the -manual_detention parameter.

You can set the -manual_detention parameter to one of the following values:

• on. The search head cluster member enters detention and does not accept any new searches. It also does not receive replicated search artifacts from other members of the cluster. The search head continues to perform other duties associated with search head clustering, such as voting for a captain.

• off. The search head cluster member accepts new searches, replicates search artifacts, and performs duties associated with search head clustering. This is the default setting.

For example:

splunk edit shcluster-config -manual_detention on

splunk edit shcluster-config -manual_detention off

The search head must be in the "up" state before you put it in detention. Verify the state of the search head before you attempt to put it in manual detention.

For information on monitoring the status of a clustered search head, see Distributed Search Dashboards

Put a search head cluster member into detention via the REST endpoint

You can use the REST endpoint shcluster/member/control/control/set_manual_detention to put a search head cluster member into manual detention.

For details, see the REST API documentation for shcluster/member/control/control/set_manual_detention.

You can also use the shcluster/config/config REST endpoint to put a search head cluster member in manual detention by sending the request to any other node in the cluster.

For an example, see the REST API documentation for shcluster/config/config .

Put a search head cluster member into detention via the server.conf file

To put a search head into manual detention, you can modify the manual_detention attribute in the [shclustering] stanza of the search head's server.conf file. You set the value to on. For example:

[shclustering]
disabled = 0
mgmt_uri = https://tsen-centos62x64-5:8089
id = C09EC4A9-8426-46F3-8385-693998B1EA5E
manual_detention = on

In order for changes to take effect, you must restart the search head cluster member when you use the server.conf file to put it into detention.

See the documentation for cluster configuration in the server.conf files here: search head clustering configuration.