Distributed Deployment Manual

Configure distributed search

Distributed search is available on every Splunk server, with the exception of forwarders. This means that every Splunk server can function as a search head to a specified group of indexers, referred to as search peers. The distributed search capability is enabled by default.

To activate distributed search, a Splunk instance that you designate as a search head just needs to add search peers. Ordinarily, you do this by specifying each search peer manually. A number of other configuration options are available, but ordinarily you do not need to alter their default values.

When configuring Splunk instances as search heads or search peers, keep this key distinction in mind:

• A search head must maintain a list of search peers, or it will have nothing to search on. A dedicated search head does not have additional data inputs, beyond the default ones.
• A search peer must have specified data inputs, or it will have nothing to index. A search peer does not maintain a list of other search peers.

These roles are not necessarily distinct. A Splunk instance can, and frequently does, function simultaneously as both a search head and a search peer.

Configuration overview

You set up distributed search on a search head using any of these methods:

• Splunk Web
• Splunk CLI
• The distsearch.conf configuration file

Splunk Web is the recommended method for most purposes.

The configuration happens on the designated search head. The main step is to specify the search head's search peers. The distributed search capability itself is already enabled by default.

Important: Before an indexer can function as a search peer, you must change its password from the default "changeme". Otherwise, the search head will not be able to authenticate against it.

Note: Starting with Splunk version 4.3, you can use an IPv6 address when specifying a search peer manually. For more information, see "Configure Splunk for IPv6" in the Admin manual.

Aside from changing the password, no configuration is generally necessary on the search peers. Access to the peers is controllable through public key authentication. However, you do need to perform some configuration on the search peers if you mount the knowledge bundles, so that the peers can access them. See "Mount the knowledge bundle" for details.

Use Splunk Web

You can add search peers manually (the usual case) or by enabling automatic discovery.

Add search peers manually

To specify search peers manually:

1. Log into Splunk Web and click Manager.

2. Click Distributed search in the Deployment area.

3. On the Search peers line, select Add new.

4. Specify the search peer, along with any authentication settings.

5. Click Save.

Add search peers automatically, or otherwise configure distributed search

To configure automatic discovery, you need to go into Distributed search setup on both the search head and the search peers. You can configure other settings there as well.

On the search head:

1. Log into Splunk Web and click Manager.

2. Click Distributed search in the Deployment area.

3. Click Distributed search setup.

4. Specify "Yes" for the option: "Automatically add other Splunk servers?".

5. Change any other settings as needed and click Save.

On each search peer:

1. Log into Splunk Web and click Manager.

2. Click Distributed search in the Deployment area.

3. Click Distributed search setup.

4. Specify "Yes" for the option: "Broadcast to other Splunk servers?"

5. Change any other settings as needed and click Save.

Use the CLI

Follow these instructions to enable distributed search via Splunk's CLI.

To use Splunk's CLI, navigate to the $SPLUNK_HOME/bin/ directory and use the ./splunk command.

Enable distributed search

Distributed search is enabled by default, so this step is ordinarily not required:

splunk enable dist-search -auth admin:password

Add a search peer manually

Use the splunk add search-server command to add a search peer. When you run this command, make sure you specify the splunkd management port of the peer server. By default, this is 8089, although it might be different for your deployment.

Be sure to provide credentials for both the local and the remote machines. Use the -auth flag for your local credentials and the -remoteUsername and -remotePassword flags to specify the remote credentials (in this example, for search peer 10.10.10.10):

splunk add search-server -host 10.10.10.10:8089 -auth admin:password -remoteUsername admin -remotePassword passremote

Specify automatic discovery

splunk enable discoverable -auth admin:password

Each of these commands will elicit a response from the server indicating success and the need to restart the server for the changes to take effect.

Edit distsearch.conf

In most cases, the settings available through Splunk Web provide sufficient options for configuring distributed search environments. Some advanced configuration settings, however, are only available through distsearch.conf. Edit this file in $SPLUNK_HOME/etc/system/local/, or your own custom application directory in $SPLUNK_HOME/etc/apps/.

For the detailed specification and examples, see distsearch.conf.

For more information on configuration files in general, see "About configuration files".

Note: As of 4.1, the blacklistNames and blacklistURLs attributes no longer have any effect on distributed search behavior.

Distribute the key files

If you add search peers via Splunk Web or the CLI, Splunk automatically handles authentication. However, if you add peers by editing distsearch.conf, you must distribute the key files manually.

After enabling distributed search on a Splunk instance (and restarting the instance), you will find the keys in this location: $SPLUNK_HOME/etc/auth/distServerKeys/

Distribute the file $SPLUNK_HOME/etc/auth/distServerKeys/trusted.pem on the search head to $SPLUNK_HOME/etc/auth/distServerKeys/<searchhead_name>/trusted.pem on the indexer nodes.

Support for keys from multiple Splunk instances

Any number of Splunk instances can have their certificates stored on other instances for authentication. The instances can store keys in $SPLUNK_HOME/etc/auth/distServerKeys/<peername>/trusted.pem

For example, if you have Splunk search heads A and B and they both need to search Splunk index node C, do the following:

1. On C, create $SPLUNK_HOME/etc/auth/distServerKeys/A/ and etc/auth/distServerKeys/B/.

2. Copy A's trusted.pem to $SPLUNK_HOME/etc/auth/distServerKeys/A/ and B's trusted.pem to $SPLUNK_HOME/etc/auth/distServerKeys/B/.

3. Restart C.

Limit the knowledge bundle size

The knowledge bundle is the data that the search head replicates and distributes to each search peer to enable its searches. For information on the contents and purpose of this bundle, see "What search heads send to search peers".

The knowledge bundle can grow quite large, as, by default, it includes nearly the entire contents of all the search head's apps. To limit the size of the bundle, you can create a replication whitelist. To do this, edit distsearch.conf and specify a [replicationWhitelist] stanza:

[replicationWhitelist]
<name> = <whitelist_regex> 
...

All files that satisfy the whitelist regex will be included in the bundle that the search head distributes to its search peers. If multiple regex's are specified, the bundle will include the union of those files.

In this example, the knowledge bundle will include all files with extensions of either ".conf" or ".spec":

[replicationWhitelist]
allConf = *.conf
allSpec = *.spec

The names, such as allConf and allSpec, are used only for layering. That is, if you have both a global and a local copy of distsearch.conf, the local copy can be configured so that it overrides only one of the regex's. For instance, assume that the example shown above is the global copy. Assume you then specify a whitelist in your local copy like this:

[replicationWhitelist]
allConf = *.foo.conf

The two conf files will be layered, with the local copy taking precedence. Thus, the search head will distribute only files that satisfy these two regex's:

allConf = *.foo.conf
allSpec = *.spec

For more information on attribute layering in configuration files, see "Attribute precedence" in the Admin manual.

You can also create a replication blacklist, using the [replicationBlacklist] stanza. The blacklist takes precedence over any whitelist. See distsearch.conf in the Admin manual for details.

As an alternative to replicating and distributing a knowledge bundle, large or small, to search peers, you can mount the knowledge bundle on shared storage. For more information, read "Mount the knowledge bundle".

Manage distributed server names

The name of each search head and search peer is determined by its serverName attribute, specified in server.conf. serverName defaults to the server's machine name.

In a distributed search cluster, all nodes must have unique names. The serverName has three specific uses:

• For authenticating search heads. When search peers are authenticating a search head, they look for the search head's key file in /etc/auth/distServerKeys/<searchhead_name>/trusted.pem.
• For identifying search peers in search queries. serverName is the value of the splunk_server field that you specify when you want to query a specific node. See "Search across one or more distributed search peers" in the User manual.
• For identifying search peers in search results. serverName gets reported back in the splunk_server field.

Note: serverName is not used when adding search peers to a search head. In that case, you identify the search peers through their domain names or IP addresses.

The only reason to change serverName is if you have multiple instances of Splunk residing on a single machine, and they're participating in the same distributed search cluster. In that case, you'll need to change serverName to distinguish them.

Remove a search peer

You can remove a search peer from a search head through the Distributed Search page on Manager. However, this only removes the search peer entry from the search head; it does not remove the search head key from the search peer. In most cases, this is not a problem and no further action is needed.

If you need to disable the trust relationship between a search peer and a search head, you can delete the search-head-specific trusted.pem file from the directory $SPLUNK_HOME/etc/auth/distServerKeys/<searchhead_name>. It's unlikely that you'll need to do this.