Configure distributed search

Configure distributed search

Distributed search is available on every Splunk server, with the exception of forwarders. This means that every Splunk server can function as a search head to a specified group of indexers, referred to as search peers. The distributed search capability is enabled by default.

To activate distributed search, a Splunk instance that you designate as a search head just needs to add search peers. Ordinarily, you do this by specifying each search peer manually. A number of other configuration options are available, but ordinarily you do not need to alter their default values.

No configuration is necessary on the search peers themselves to make them available to search heads. Access is controllable through public key authentication.

When configuring Splunk instances as search heads or search peers, keep this key distinction in mind:

• A search head must maintain a list of search peers, or it will have nothing to search on. A dedicated search head does not have additional data inputs, beyond the default ones.
• A search peer must have specified data inputs, or it will have nothing to index. A search peer does not maintain a list of other search peers.

These roles are not necessarily distinct. A Splunk instance can, and frequently does, function simultaneously as both a search head and a search peer.

You can set up a distributed search head via Splunk Web, the Splunk CLI, or by editing the distsearch.conf configuration file. Splunk Web is the recommended configuration method for most purposes.

Use Splunk Web

The main step in configuring a distributed search head is to specify its search peers. You can add these either manually (the usual case) or by enabling automatic discovery.

Add search peers manually

To specify search peers manually:

1. Log into Splunk Web and click Manager.

2. Click Distributed search.

3. On the Search peers line, select Add new.

4. Specify the search peer, along with any authentication settings.

5. Click Save.

Add search peers automatically, or otherwise configure distributed search

To configure automatic discovery, you need to go into Distributed search setup on both the search head and the search peers. You can configure other settings there as well.

On the search head:

1. Log into Splunk Web and click Manager.

2. Click Distributed search.

3. Click Distributed search setup.

4. Specify "Yes" for the option: "Automatically add other Splunk servers?".

5. Change any other settings as needed and click Save.

On each search peer:

1. Log into Splunk Web and click Manager.

2. Click Distributed search.

3. Click Distributed search setup.

4. Specify "Yes" for the option: "Broadcast to other Splunk servers?"

5. Change any other settings as needed and click Save.

Use the CLI

Follow these instructions to enable distributed search via Splunk's CLI.

To use Splunk's CLI, navigate to the $SPLUNK_HOME/bin/ directory and use the ./splunk command.

Enable distributed search

Distributed search is enabled by default, so this step is ordinarily not required:

splunk enable dist-search -auth admin:changeme

Add a search peer manually

Use the splunk add search-server command to add a search peer. When you run this command, make sure you specify the splunkd management port of the peer server. By default, this is 8089, although it might be different for your deployment.

Be sure to provide credentials for both the local and the remote machines. Use the -auth flag for your local credentials and the -remoteUsername and -remotePassword flags to specify the remote credentials (in this example, for search peer 10.10.10.10):

splunk add search-server -host 10.10.10.10:8089 -auth admin:changeme -remoteUsername admin -remotePassword changeme

Specify automatic discovery

splunk enable discoverable -auth admin:changeme

Each of these commands will elicit a response from the server indicating success and the need to restart the server for the changes to take effect.

Edit distsearch.conf

In most cases, the settings available through Splunk Web provide sufficient options for configuring distributed search environments. Some advanced configuration settings, however, are only available through distsearch.conf. Edit this file in $SPLUNK_HOME/etc/system/local/, or your own custom application directory in $SPLUNK_HOME/etc/apps/.

For the detailed specification and examples, see distsearch.conf.

For more information on configuration files in general, see "About configuration files".

Note: As of 4.1, the blacklistNames and blacklistURLs attributes no longer have any effect on distributed search behavior.

Distribute the key files

If you add search peers via Splunk Web or the CLI, Splunk automatically handles authentication. However, if you add peers by editing distsearch.conf, you must distribute the key files manually.

After enabling distributed search on a Splunk instance (and restarting the instance), you will find the keys in this location: $SPLUNK_HOME/etc/auth/distServerKeys/

Distribute the file $SPLUNK_HOME/etc/auth/distServerKeys/trusted.pem on the search head to $SPLUNK_HOME/etc/auth/distServerKeys/<searchhead_name>/trusted.pem on the indexer nodes.

Support for keys from multiple Splunk instances

Any number of Splunk instances can have their certificates stored on other instances for authentication. The instances can store keys in $SPLUNK_HOME/etc/auth/distServerKeys/<peername>/trusted.pem

For example, if you have Splunk search heads A and B and they both need to search Splunk index node C, do the following:

1. On C, create $SPLUNK_HOME/etc/auth/distServerKeys/A/ and etc/auth/distServerKeys/B/.

2. Copy A's trusted.pem to $SPLUNK_HOME/etc/auth/distServerKeys/A/ and B's trusted.pem to $SPLUNK_HOME/etc/auth/distServerKeys/B/.

3. Restart C.

Limit distributed bundle size

The distributed knowledge bundle is the data that the search head replicates to each search peer to enable its searches. For information on the contents and purpose of this bundle, see "What search heads send to search peers".

To limit the size of the distributed bundle, you can create a replication whitelist. To do this, edit distsearch.conf and specify a [replicationWhitelist] stanza:

[replicationWhitelist]
<name> = <whitelist_regex> 
...

All files that satisfy the whitelist regex will be included in the bundle that the search head distributes to its search peers. If multiple regex's are specified, the bundle will inlucde the union of those files.

In this example, the knowledge bundle will include all files with extensions of either ".conf" or ".spec":

[replicationWhitelist]
allConf = *.conf
allSpec = *.spec

The names, such as allConf and allSpec, are used only for layering. That is, if you have both a global and a local copy of distsearch.conf, the local copy can be configured so that it overrides only one of the regex's. For instance, assume that the example shown above is the global copy. Assume you then specify a whitelist in your local copy like this:

[replicationWhitelist]
allConf = *.foo.conf

The two conf files will be layered, with the local copy taking precedence. Thus, the search head will distribute only files that satisfy these two regex's:

allConf = *.foo.conf
allSpec = *.spec

For more information on attribute layering in configuration files, see "Attribute precedence" in this manual.

Manage distributed server names

The name of each search head and search peer is determined by its serverName attribute, specified in server.conf. serverName defaults to the server's machine name.

In a distributed search cluster, all nodes must have unique names. The serverName has three specific uses:

• For authenticating search heads. When search peers are authenticating a search head, they look for the search head's key file in /etc/auth/distServerKeys/<searchhead_name>/trusted.pem.
• For identifying search peers in search queries. serverName is the value of the splunk_server field that you specify when you want to query a specific node. See "Search across one or more distributed search peers" in the User manual.
• For identifying search peers in search results. serverName gets reported back in the splunk_server field.

Note: serverName is not used when adding search peers to a search head. In that case, you identify the search peers through their domain names or IP addresses.

The only reason to change serverName is if you have multiple instances of Splunk residing on a single machine, and they're participating in the same distributed search cluster. In that case, you'll need to change serverName to distinguish them.

Disable trust relationship between a search peer and a search head

If you need to disable access to a search peer from a particular search head, you can delete the search-head-specific trusted.pem file from the directory $SPLUNK_HOME/etc/auth/distServerKeys/<searchhead_name>. It's unlikely that you'll need to do this.

• Was this topic useful?
• Post a comment