Connect the search heads in clusters to search peers

Before the search heads in the cluster can run searches, they need to know the identity of their indexers, or search peers. All members of a cluster must have access to the same set of search peers.

How the search heads find out about their search peers depends on whether the search head cluster is part of an indexer cluster. There are two scenarios to consider:

• The search head cluster will be running against an indexer cluster.
• The search head cluster will be running against individual, non-clustered indexers.

Important: Cluster members cannot distribute searches to other cluster members. In other words, a cluster member cannot be a search peer of the cluster.

Search head cluster with indexer cluster

If the search head cluster is connected to an indexer cluster, the manager node on the indexer cluster provides the search heads with a list of peer nodes to search against.

Once you configure the search head cluster members so that they participate in the indexer cluster, you do not need to perform any further configuration for the search heads to know their search peers. See Integrate the search head cluster with an indexer cluster.

Even if you do not need the benefits of index replication, you can still take advantage of this simple approach to configuring the set of search peers. Just incorporate your set of indexers into an indexer cluster with a replication factor of 1. This topology also provides numerous other benefits from a management perspective. See Use indexer clusters to scale indexing in the Managing Indexers and Clusters of Indexers manual.

Search head cluster with non-clustered indexers

You can add non-clustered search peers in two ways:

• Add the search peers to each member individually.
• Add the search peers to one member and let the cluster replicate the peer configurations to all other cluster members. This is known as search peer replication.

Before Splunk Enterprise 6.4, only the first method was available. You had to add the search peers to each individual member. Starting with 6.4, you can add the search peers to just a single member and let the cluster replicate the peer configurations to the other members.

The replication method is usually preferable, for several reasons:

• It is simpler and faster.
• It ensures that all members have access to all peers.
• If you later add a new member to the cluster, it automatically gets the set of peers.

The main circumstance where you might prefer to add peers to individual members is if you already have a cluster and you have automated the process of adding search peers to each member.

You can switch to the replication method at any time. Peers already added individually will remain in the configuration. If you add a new member later, it will get the full set of peers, no matter how they were originally added to the cluster.

Note: The replication method does not use the configuration replication method described in Configuration updates that the cluster replicates. Instead, it uses a Raft state machine to replicate the changes to all active members. With this method, all active members receive the add request at the same time, ensuring that all members gain access to the same set of search peers.

Replicate the search peers across the cluster

1. Enable search peer replication on each member.

In each member's server.conf file, configure the [raft_statemachine] stanza as follows:

[raft_statemachine]
disabled = false
replicate_search_peers = true

2. Restart each search head cluster member.

3. Use the CLI to add the search peers to one member. It does not matter which member you perform this on.

On one member, run the following command, one time for each search peer:

splunk add search-server <scheme>://<host>:<port> -auth <user>:<password> -remoteUsername <user> -remotePassword <passremote>

Note the following:

• <scheme> is the URI scheme for accessing the search peer: "http" or "https".
• <host> is the host name or IP address of the search peer's host machine.
• <port> is the management port of the search peer.
• -auth provides credentials for the member.
• -remoteUsername and -remotePassword provide credentials for the search peer. The remote credentials must be for an admin-level user on the search peer.

For example:

splunk add search-server https://192.168.1.1:8089 -auth admin:password -remoteUsername admin -remotePassword passremote

When you add a search peer to one cluster member, the cluster quickly replicates the operation to the other members. The members will then commit the change together.

Important: To add a peer through replication, you need a healthy cluster. Captaincy should remain with the same member until all active members have successfully committed the change. If you encounter a problem and the change does not get committed with the current captain, remediation is simple: Just rerun the splunk add search-server command.

4. Repeat the splunk add search-server command for each search peer.

Note: You can also use replication to remove search peers from the cluster members. See Remove a search peer via the CLI.

Add search peers to each member individually

To add the search peers individually to each search head, use the CLI. On each search head, invoke the splunk add search-server command for each search peer that you want to add:

splunk add search-server <scheme>://<host>:<port> -auth <user>:<password> -remoteUsername <user> -remotePassword <passremote>

You must repeat this procedure on each search head, for each search peer. For example, on a three member cluster, with five search peers, you must run this command a total of 15 times.

Caution: All search heads must use the same set of search peers.

Add search peers through Splunk Web

In addition to the CLI, you can add search peers through Splunk Web:

1. Unhide the hidden settings on the search head, as described in The Settings menu.

2. Follow the instructions in Use Splunk Web.

If you have enabled search peer replication, you add the search peers to only one of the cluster members. If you have not enabled search peer replication, you must add them to each cluster member.

Add search peers by directly editing distsearch.conf

If you are not using search peer replication, you can add search peers by directly editing distsearch.conf and distributing the configuration file via the deployer. This method requires that you also manually distribute the key file from each search head to each search peer. See Edit distsearch.conf.

Because of the need to manually distribute key files, this method is not compatible with search peer replication.

Forward search head data to the search peers

It is considered a best practice to forward all search head internal data to the search peer (indexer) layer. After you connect the search heads to the search peers, follow the instructions in Best practice: Forward search head data to the indexer layer.