Quarantine a search peer

You can quarantine a search peer to prevent it from partaking in future searches. This is of value if the peer is experiencing problems, for example, due to a bad disk or network card. It can also be useful to quarantine a search peer while you upgrade it.

By quarantining, instead of stopping, a bad search peer, you can perform live troubleshooting on the peer.

You can override a quarantine for a specific search, if necessary. See How to override a quarantine.

What happens when you quarantine a search peer

The quarantine operation affects only the relationship between a search head and a search peer. You quarantine the search peer from the search head.

When you quarantine a search peer, the search head stops sending search requests to the search peer. This action prevents the search peer from taking part in new searches. The search peer, however, continues to service any currently running searches. The search peer also continues to receive and index incoming data in its role as an indexer.

If you need to fully halt the activities of the indexer, you must bring it down.

If the search peer is a member of an indexer cluster, it continues to replicate data from other peer nodes, in addition to continuing to index incoming data. The cluster takes no action to compensate for the loss of search capability on the search peer. If you need the cluster to maintain search capability across all buckets, you can take the peer offline temporarily instead of quarantining it. When you take a peer ofline, the cluster reassigns primary bucket copies to other peers. See Take a peer offline in the Managing Indexers and Clusters of Indexers manual.

How to quarantine a search peer

To quarantine a search peer, run this CLI command from the search head:

splunk edit search-server -auth <user>:<password> <host>:<port> -action quarantine

Note the following:

• Use the -auth flag to provide credentials for the search head only.
• <host> is the host name or IP address of the search peer's host machine.
• <port> is the management port of the search peer.

For example:

splunk edit search-server -auth admin:password 10.10.10.10:8089 -action quarantine

In a search head cluster, this command affects only the search head that it is run on. To quarantine a peer for all cluster members, you must run this command on each member.

You can also quarantine a search peer through the Search peers page on the search head's Splunk Web. See View search peer status in Settings.

How to unquarantine a search peer

To remove a search peer from quarantine, run this command from the search head:

splunk edit search-server  -auth <user>:<password> <host>:<port> -action unquarantine

Note the following:

• Use the -auth flag to provide credentials for the search head only.
• <host> is the host name or IP address of the search peer's host machine.
• <port> is the management port of the search peer.

For example:

splunk edit search-server -auth admin:password 10.10.10.10:8089 -action unquarantine

How to override a quarantine

When a peer is quarantined, it does not ordinarily participate in searches. You can, however, override the quarantine on a search-by-search basis. To do so, the search must target the peer directly with the splunk_server field. For example:

index=_internal splunk_server=idx-tk421-03 (log_level=WARN OR log_level=ERROR)

Note: If the peer is a member of a distributed search group, you cannot override the quarantine by specifying the splunk_server_group field of its search group. You must specify the peer directly with the splunk_server field.