Handle slow search peers

A search normally continues to run until all search peers return the requested data. This can sometimes create a problem in deployments with very large numbers of search peers (100+). If one of the peers is much slower than the others in returning its portion of the data (for example, due to network issues), the searches can continue for abnormally long periods of time while awaiting the final results from that peer.

If such a situation arises and you want to trade data fidelity for search performance, you can direct the search head to end long-running searches without waiting for a slow peer to finish sending all its data. To do this, you enable the search head's [slow_peer_disconnect] stanza in limits.conf. By default, this capability is disabled. You can toggle the capability without restarting the search head.

The heuristics that determine when to disconnect a search from a slow peer are complex and tunable by means of several parameters in the [slow_peer_disconnect] stanza. If you feel the need to use this capability, contact Splunk Professional Services for guidance in adjusting the heuristics for your specific deployment needs.