Distributed Search

Connect the search heads in clusters to search peers

Before the search heads in the cluster can run searches, they need to know the identity of their indexers, or search peers. All members of a cluster must have access to the same set of search peers.

How the search heads find out about their search peers depends on whether the search head cluster is part of an indexer cluster. There are two scenarios to consider:

• The search head cluster will be running against an indexer cluster.
• The search head cluster will be running against individual, non-clustered indexers.

Important: Cluster members cannot distribute searches to other cluster members. In other words, a cluster member cannot be a search peer of the cluster.

Search head cluster with indexer cluster

If the search head cluster is connected to an indexer cluster, the master node on the indexer cluster provides the search heads with a list of peer nodes to search against.

Once you configure the search head cluster members so that they participate in the indexer cluster, you do not need to perform any further configuration for the search heads to know their search peers. See "Integrate the search head cluster with an indexer cluster".

Even if you do not need the benefits of index replication, you can still take advantage of this simple approach to configuring the set of search peers. Just incorporate your set of indexers into an indexer cluster with a replication factor of 1. This topology also provides numerous other benefits from a management perspective. See "Use indexer clusters to scale indexing" in the Managing Indexers and Clusters of Indexers manual.

Search head cluster with non-clustered indexers

If you do not have an indexer cluster, you must add the search peers individually to each search head. The easiest way to do this is through the CLI.

On each search head, invoke the splunk add search-server command for each search peer that you want to add:

splunk add search-server -host <URI>:<management_port> -auth <user>:<password> -remoteUsername <user> -remotePassword <password>

Note the following:

• Use the -host flag to specify the search peer's URI and management port.
• Provide credentials for both the local (search head) and remote (search peer) instances. Use the -auth flag for the local credentials and the -remoteUsername and -remotePassword flags for the remote credentials. The remote credentials must be for an admin-level user on the search peer.

In this example, the search peer has an IP address of 10.10.10.10, a management port of 8089, and an admin user with a password of passremote:

splunk add search-server -host 10.10.10.10:8089 -auth admin:mypassword -remoteUsername admin -remotePassword passremote

You must repeat this procedure on each search head, for each search peer. For example, with a three member cluster, with five search peers, you must run this command a total of 15 times.

Caution: All search heads must use the same set of search peers.

You can also add search peers through Splunk Web on each search head. To do this, you must first unhide the hidden settings, as described in "The Settings menu." Then follow the instructions in "Add search peers to the search head."

Forward search head data to the search peers

It is considered a best practice to forward all search head internal data to the search peer (indexer) layer. After you connect the search heads to the search peers, follow the instructions in "Best practice: Forward search head data to the indexer layer."