Add search peers to the search head

To activate distributed search, you add search peers, or indexers, to a Splunk Enterprise instance that you designate as a search head. You do this by specifying each search peer manually.

Important: A search head cannot perform a dual function as a search peer. The only exception to this rule is for the monitoring console, which functions as a "search head of search heads."

This topic describes how to connect a search head to a set of search peers.

If you need to connect multiple search heads to a set of search peers, you can repeat the process for each search head individually. However, if you require multiple search heads, the best practice is to deploy them in a search head cluster. A search head cluster can also replicate all search peers from one search head to all the other search heads in the cluster, so that you do not have to add the peers to each search head separately.

Important: Clusters establish connectivity between search heads and search peers differently from the procedures described in this topic:

• Indexer clusters automatically establish the connection between their search heads and indexers, or peer nodes. To learn how to configure search heads in indexer clusters, read Configure the search head in the Managing Indexers and Clusters of Indexers manual.
• Search head clusters have certain restrictions that you must consider when connecting search heads to search peers. See Connect the search heads in clusters to search peers.

Configuration overview

To set up the connection between a search head and its search peers, configure the search head through one of these methods:

• Splunk Web
• Splunk CLI
• The distsearch.conf configuration file

Splunk Web is the simplest method for most purposes.

The configuration occurs on the search head. For most deployments, no configuration is necessary on the search peers. Access to the peers is controlled through public key authentication.

Prerequisites

Before an indexer can function as a search peer, you must change its password from the default value. Otherwise, the search head will not be able to authenticate against it.

Use Splunk Web

Specify the search peers

To specify the search peers:

1. Log into Splunk Web on the search head and click Settings at the top of the page.

2. Click Distributed search in the Distributed Environment area.

3. Click Search peers.

4. On the Search peers page, select New.

5. Specify the search peer, along with any authentication settings.

Note: You must precede the search peer's host name or IP address with the URI scheme, either "http" or "https".

6. Click Save.

7. Repeat for each of the search head's search peers.

Configure miscellaneous distributed search settings

To configure other settings:

1. Log into Splunk Web on the search head and click Settings at the top of the page.

2. Click Distributed search in the Distributed Environment area.

3. Click Distributed search setup.

5. Change any settings as needed.

6. Click Save.

Use the CLI

To add a search peer, run this command from the search head:

splunk add search-server <scheme>://<host>:<port> -auth <user>:<password> -remoteUsername <user> -remotePassword <passremote>

Note the following:

• <scheme> is the URI scheme: "http" or "https".
• <host> is the host name or IP address of the search peer's host machine.
• <port> is the management port of the search peer.
• Use the -auth flag to provide credentials for the search head.
• Use the -remoteUsername and -remotePassword flags for the credentials for the search peer. The remote credentials must be for an admin-level user on the search peer.

For example:

splunk add search-server https://192.168.1.1:8089 -auth admin:password -remoteUsername admin -remotePassword passremote

You must run this command for each search peer that you want to add.

Edit distsearch.conf

The settings available through Splunk Web provide sufficient options for most configurations. Some advanced configuration settings, however, are only available by directly editing distsearch.conf. This section discusses only the configuration settings necessary for connecting search heads to search peers. For information on the advanced configuration options, see the distsearch.conf spec file.

Add the search peers

To connect the search peers:

1. On the search head, create or edit a distsearch.conf file in $SPLUNK_HOME/etc/system/local.

2. Add the search peers to the servers setting under the [distributedSearch] stanza. Specify the peers as a set of comma-separated values (host names or IP addresses with management ports). For example:

[distributedSearch]
servers = https://192.168.1.1:8089,https://192.168.1.2:8089

Note: You must precede the host name or IP address with the URI scheme, either "http" or "https".

3. Restart the search head.

Distribute the key files

If you add search peers via Splunk Web or the CLI, Splunk Enterprise automatically configures authentication. However, if you add peers by editing distsearch.conf, you must distribute the key files manually. After adding the search peers and restarting the search head, as described above:

1. Copy the file $SPLUNK_HOME/etc/auth/distServerKeys/trusted.pem from the search head to $SPLUNK_HOME/etc/auth/distServerKeys/<searchhead_name>/trusted.pem on each search peer.

The <searchhead_name> is the search head's serverName, specified in server.conf.

2. Restart each search peer.

Authentication of multiple search heads from a single peer

Multiple search heads can search across a single peer. The peer must store a copy of each search head's certificate.

The search peer stores the search head keys in directories with the specification $SPLUNK_HOME/etc/auth/distServerKeys/<searchhead_name>.

For example, if you have two search heads, named A and B, and they both need to search one particular search peer, do the following:

1. On the search peer, create the directories $SPLUNK_HOME/etc/auth/distServerKeys/A/ and $SPLUNK_HOME/etc/auth/distServerKeys/B/.

2. Copy A's trusted.pem file to $SPLUNK_HOME/etc/auth/distServerKeys/A/ and B's trusted.pem to $SPLUNK_HOME/etc/auth/distServerKeys/B/.

3. Restart the search peer.

Group the search peers

You can group search peers into distributed search groups. This allows you to target searches to subsets of search peers. See Create distributed search groups.

View search peer status

See View search peer status in Settings.