
Configure distributed search
Distributed search is available on every Splunk server, with the exception of forwarders. This means that every Splunk server can function as a search head to a specified group of indexers, referred to as search peers. The distributed search capability is enabled by default.
To activate distributed search, a Splunk instance that you designate as a search head just needs to add search peers. Ordinarily, you do this by specifying each search peer manually. A number of other configuration options are available, but ordinarily you do not need to alter their default values.
When configuring Splunk instances as search heads or search peers, keep this key distinction in mind:
- A search head must maintain a list of search peers, or it will have nothing to search on. A dedicated search head does not have additional data inputs, beyond the default ones.
- A search peer must have specified data inputs, or it will have nothing to index. A search peer does not maintain a list of other search peers.
These roles are not necessarily distinct. A Splunk instance can, and frequently does, function simultaneously as both a search head and a search peer.
Configuration overview
You set up distributed search on a search head using any of these methods:
- Splunk Web
- Splunk CLI
- The
distsearch.conf
configuration file
Splunk Web is the recommended method for most purposes.
The configuration happens on the designated search head. The main step is to specify the search head's search peers. The distributed search capability itself is already enabled by default.
Important: Before an indexer can function as a search peer, you must change its password from the default "changeme". Otherwise, the search head will not be able to authenticate against it.
Note: Starting with Splunk version 4.3, you can use an IPv6 address when specifying a search peer manually. For more information, see "Configure Splunk for IPv6" in the Admin manual.
Aside from changing the password, no configuration is generally necessary on the search peers. Access to the peers is controllable through public key authentication. However, you do need to perform some configuration on the search peers if you mount the knowledge bundles, so that the peers can access them. See "Mount the knowledge bundle" for details.
Use Splunk Web
You can add search peers manually (the usual case) or by enabling automatic discovery.
Add search peers manually
To specify search peers manually:
1. Log into Splunk Web and click Manager.
2. Click Distributed search in the Deployment area.
3. On the Search peers line, select Add new.
4. Specify the search peer, along with any authentication settings.
5. Click Save.
Add search peers automatically, or otherwise configure distributed search
To configure automatic discovery, you need to go into Distributed search setup on both the search head and the search peers. You can configure other settings there as well.
On the search head:
1. Log into Splunk Web and click Manager.
2. Click Distributed search in the Deployment area.
3. Click Distributed search setup.
4. Specify "Yes" for the option: "Automatically add other Splunk servers?".
5. Change any other settings as needed and click Save.
On each search peer:
1. Log into Splunk Web and click Manager.
2. Click Distributed search in the Deployment area.
3. Click Distributed search setup.
4. Specify "Yes" for the option: "Broadcast to other Splunk servers?"
5. Change any other settings as needed and click Save.
Use the CLI
Follow these instructions to enable distributed search via Splunk's CLI.
To use Splunk's CLI, navigate to the $SPLUNK_HOME/bin/
directory and use the ./splunk
command.
Enable distributed search
Distributed search is enabled by default, so this step is ordinarily not required:
splunk enable dist-search -auth admin:password
Add a search peer manually
Use the splunk add search-server
command to add a search peer. When you run this command, make sure you specify the splunkd management port of the peer server. By default, this is 8089
, although it might be different for your deployment.
Be sure to provide credentials for both the local and the remote machines. Use the -auth
flag for your local credentials and the -remoteUsername
and -remotePassword
flags to specify the remote credentials (in this example, for search peer 10.10.10.10
):
splunk add search-server -host 10.10.10.10:8089 -auth admin:password -remoteUsername admin -remotePassword passremote
Specify automatic discovery
splunk enable discoverable -auth admin:password
Each of these commands will elicit a response from the server indicating success and the need to restart the server for the changes to take effect.
Edit distsearch.conf
In most cases, the settings available through Splunk Web provide sufficient options for configuring distributed search environments. Some advanced configuration settings, however, are only available through distsearch.conf
. Edit this file in $SPLUNK_HOME/etc/system/local/
, or your own custom application directory in $SPLUNK_HOME/etc/apps/
.
For the detailed specification and examples, see distsearch.conf.
For more information on configuration files in general, see "About configuration files".
Note: As of 4.1, the blacklistNames
and blacklistURLs
attributes no longer have any effect on distributed search behavior.
Distribute the key files
If you add search peers via Splunk Web or the CLI, Splunk automatically handles authentication. However, if you add peers by editing distsearch.conf
, you must distribute the key files manually.
After enabling distributed search on a Splunk instance (and restarting the instance), you will find the keys in this location: $SPLUNK_HOME/etc/auth/distServerKeys/
Distribute the file $SPLUNK_HOME/etc/auth/distServerKeys/trusted.pem
on the search head to $SPLUNK_HOME/etc/auth/distServerKeys/<searchhead_name>/trusted.pem
on the indexer nodes.
Support for keys from multiple Splunk instances
Any number of Splunk instances can have their certificates stored on other instances for authentication.
The instances can store keys in $SPLUNK_HOME/etc/auth/distServerKeys/<peername>/trusted.pem
For example, if you have Splunk search heads A and B and they both need to search Splunk index node C, do the following:
1. On C, create $SPLUNK_HOME/etc/auth/distServerKeys/A/
and etc/auth/distServerKeys/B/
.
2. Copy A's trusted.pem
to $SPLUNK_HOME/etc/auth/distServerKeys/A/
and B's trusted.pem
to $SPLUNK_HOME/etc/auth/distServerKeys/B/
.
3. Restart C.
Limit the knowledge bundle size
The knowledge bundle is the data that the search head replicates and distributes to each search peer to enable its searches. For information on the contents and purpose of this bundle, see "What search heads send to search peers".
The knowledge bundle can grow quite large, as, by default, it includes nearly the entire contents of all the search head's apps. To limit the size of the bundle, you can create a replication whitelist. To do this, edit distsearch.conf
and specify a [replicationWhitelist]
stanza:
[replicationWhitelist] <name> = <whitelist_regex> ...
All files that satisfy the whitelist regex will be included in the bundle that the search head distributes to its search peers. If multiple regex's are specified, the bundle will include the union of those files.
In this example, the knowledge bundle will include all files with extensions of either ".conf" or ".spec":
[replicationWhitelist] allConf = *.conf allSpec = *.spec
The names, such as allConf and allSpec, are used only for layering. That is, if you have both a global and a local copy of distsearch.conf
, the local copy can be configured so that it overrides only one of the regex's. For instance, assume that the example shown above is the global copy. Assume you then specify a whitelist in your local copy like this:
[replicationWhitelist] allConf = *.foo.conf
The two conf files will be layered, with the local copy taking precedence. Thus, the search head will distribute only files that satisfy these two regex's:
allConf = *.foo.conf allSpec = *.spec
For more information on attribute layering in configuration files, see "Attribute precedence" in the Admin manual.
You can also create a replication blacklist, using the [replicationBlacklist]
stanza. The blacklist takes precedence over any whitelist. See distsearch.conf in the Admin manual for details.
As an alternative to replicating and distributing a knowledge bundle, large or small, to search peers, you can mount the knowledge bundle on shared storage. For more information, read "Mount the knowledge bundle".
Manage distributed server names
The name of each search head and search peer is determined by its serverName
attribute, specified in server.conf. serverName
defaults to the server's machine name.
In a distributed search cluster, all nodes must have unique names. The serverName
has three specific uses:
- For authenticating search heads. When search peers are authenticating a search head, they look for the search head's key file in
/etc/auth/distServerKeys/<searchhead_name>/trusted.pem
. - For identifying search peers in search queries.
serverName
is the value of thesplunk_server
field that you specify when you want to query a specific node. See "Search across one or more distributed search peers" in the User manual. - For identifying search peers in search results.
serverName
gets reported back in thesplunk_server
field.
Note: serverName
is not used when adding search peers to a search head. In that case, you identify the search peers through their domain names or IP addresses.
The only reason to change serverName
is if you have multiple instances of Splunk residing on a single machine, and they're participating in the same distributed search cluster. In that case, you'll need to change serverName
to distinguish them.
Remove a search peer
You can remove a search peer from a search head through the Distributed Search page on Manager. However, this only removes the search peer entry from the search head; it does not remove the search head key from the search peer. In most cases, this is not a problem and no further action is needed.
If you need to disable the trust relationship between a search peer and a search head, you can delete the search-head-specific trusted.pem
file from the directory $SPLUNK_HOME/etc/auth/distServerKeys/<searchhead_name>
. It's unlikely that you'll need to do this.
PREVIOUS Install a dedicated search head |
NEXT Mount the knowledge bundle |
This documentation applies to the following versions of Splunk® Enterprise: 4.3, 4.3.1, 4.3.2, 4.3.3, 4.3.4, 4.3.5, 4.3.6, 4.3.7
Feedback submitted, thanks!