Splunk® Enterprise

Distributed Deployment Manual

Download manual as PDF

Splunk Enterprise version 5.0 reached its End of Life on December 1, 2017. Please see the migration information.
This documentation does not apply to the most recent version of Splunk. Click here for the latest version.
Download topic as PDF

Configure distributed search

Distributed search is available on every Splunk server, with the exception of forwarders. This means that every Splunk server can function as a search head to a specified group of indexers, referred to as search peers. The distributed search capability is enabled by default.

To activate distributed search, a Splunk instance that you designate as a search head just needs to add search peers. Ordinarily, you do this by specifying each search peer manually. A number of other configuration options are available, but ordinarily you do not need to alter their default values.

When configuring Splunk instances as search heads or search peers, keep this key distinction in mind:

  • A search head must maintain a list of search peers, or it will have nothing to search on. A dedicated search head does not have additional data inputs, beyond the default ones.
  • A search peer must have specified data inputs, or it will have nothing to index. A search peer does not maintain a list of other search peers.

These roles are not necessarily distinct. A Splunk instance can, and frequently does, function simultaneously as both a search head and a search peer.

Important: Clusters also use search heads to search across the set of indexers, or peer nodes. You deploy and configure search heads very differently when they are part of a cluster. To learn more about configuring search heads in clusters, read "Configure the search head" in the Managing Indexers and Clusters Manual.

Configuration overview

You set up distributed search on a search head using any of these methods:

  • Splunk Web
  • Splunk CLI
  • The distsearch.conf configuration file

Splunk Web is the recommended method for most purposes.

The configuration happens on the designated search head. The main step is to specify the search head's search peers. The distributed search capability itself is already enabled by default.

Important: Before an indexer can function as a search peer, you must change its password from the default "changeme". Otherwise, the search head will not be able to authenticate against it.

Aside from changing the password, no configuration is generally necessary on the search peers. Access to the peers is controllable through public key authentication. However, you do need to perform some configuration on the search peers if you mount the knowledge bundles, so that the peers can access them. See "Mount the knowledge bundle" for details.

Use Splunk Web

Specify the search peers

To specify the search peers:

1. Log into Splunk Web on the search head and click Manager at the top of the page.

2. Click Distributed search in the Distributed Environment area.

3. Click Search peers.

4. On the Search peers page, select New.

5. Specify the search peer, along with any authentication settings.

6. Click Save.

7. Repeat for each of the search head's search peers.

Configure miscellaneous distributed search settings

To configure other settings:

1. Log into Splunk Web on the search head and click Manager at the top of the page.

2. Click Distributed search in the Distributed Environment area.

3. Click Distributed search setup.

5. Change any settings as needed.

6. Click Save.

Use the CLI

Follow these instructions to enable distributed search via Splunk's CLI.

To use Splunk's CLI, navigate to the $SPLUNK_HOME/bin/ directory and use the ./splunk command.

Enable distributed search

Distributed search is enabled by default, so this step is ordinarily not required:

splunk enable dist-search -auth admin:password

Add a search peer

Use the splunk add search-server command to add a search peer. When you run this command, make sure you specify the splunkd management port of the peer server. By default, this is 8089, although it might be different for your deployment.

Be sure to provide credentials for both the local and the remote machines. Use the -auth flag for your local credentials and the -remoteUsername and -remotePassword flags to specify the remote credentials (in this example, for search peer 10.10.10.10):

splunk add search-server -host 10.10.10.10:8089 -auth admin:password -remoteUsername admin -remotePassword passremote

A message indicates success.

Edit distsearch.conf

In most cases, the settings available through Splunk Web provide sufficient options for configuring distributed search environments. Some advanced configuration settings, however, are only available through distsearch.conf. Edit this file in $SPLUNK_HOME/etc/system/local/, or your own custom application directory in $SPLUNK_HOME/etc/apps/.

For the detailed specification and examples, see distsearch.conf.

For more information on configuration files in general, see "About configuration files".

Distribute the key files

If you add search peers via Splunk Web or the CLI, Splunk automatically handles authentication. However, if you add peers by editing distsearch.conf, you must distribute the key files manually.

After enabling distributed search on a Splunk instance (and restarting the instance), you will find the keys in this location: $SPLUNK_HOME/etc/auth/distServerKeys/

Distribute the file $SPLUNK_HOME/etc/auth/distServerKeys/trusted.pem on the search head to $SPLUNK_HOME/etc/auth/distServerKeys/<searchhead_name>/trusted.pem on the indexer nodes.

Support for keys from multiple Splunk instances

Any number of Splunk instances can have their certificates stored on other instances for authentication. The instances can store keys in $SPLUNK_HOME/etc/auth/distServerKeys/<peername>/trusted.pem

For example, if you have Splunk search heads A and B and they both need to search Splunk index node C, do the following:

1. On C, create $SPLUNK_HOME/etc/auth/distServerKeys/A/ and etc/auth/distServerKeys/B/.

2. Copy A's trusted.pem to $SPLUNK_HOME/etc/auth/distServerKeys/A/ and B's trusted.pem to $SPLUNK_HOME/etc/auth/distServerKeys/B/.

3. Restart C.

Modify the knowledge bundle

The knowledge bundle is the data that the search head replicates and distributes to each search peer to enable its searches. For information on the contents and purpose of this bundle, see "What search heads send to search peers".

The knowledge bundle consists of a set of files that the search peers ordinarily need in order to perform their searches. You can, if necessary, modify this set of files. The main reasons for modifying the set of files are if:

  • As an app developer, you want to customize the files for the needs of your app. This case usually involves manipulating the replication whitelist. You can also use a replication blacklist for this purpose.
  • As an admin, you need to limit the size of the knowledge bundle. This case is somewhat unusual, because Splunk Enterprise uses delta-based replication to keep the bundle compact, with the search head usually only replicating the changed portion of the bundle to its search peers. This case requires that you identify unnecessary files and filter them out with a replication blacklist. It is also possible, although less common, to use a whitelist for this purpose.

See distsearch.conf in the Admin Manual for details on the settings discussed in this topic.

Customize the bundle for an app

The system looks at two stanzas in distsearch.conf to determine which *.conf files to include in the bundle, in this order:

1. [replicationWhitelist]

2. [replicationSettings:refineConf]

You typically only need to edit the [replicationSettings:refineConf] stanza to customize the bundle for your app, but, under rare circumstances, you might also need to modify the [replicationWhitelist] stanza.

Since the system starts by examining the [replicationWhitelist] stanza, this discussion does too.

Edit the replicationWhitelist stanza

The [replicationWhitelist] stanza in the system default version of distsearch.conf whitelists all the *.conf files that are specified in the [replicationSettings:refineConf] stanza. Therefore, to add or delete a *.conf file from the bundle, do not modify this stanza. Instead, change the set of files specified in the [replicationSettings:refineConf] stanza, as described in the next section, "Edit the replicationSettings:refineConf stanza."

The main reason for modifying the [replicationWhitelist] stanza is to include in the bundle some type of special file for use in a custom search command. This is an unusual circumstance.

If you do need to alter the whitelist, you can override the system default whitelist by creating a version of the [replicationWhitelist] stanza in $SPLUNK_HOME/etc/apps/<appname>/default/distsearch.conf:

[replicationWhitelist]
<name> = <whitelist_regex> 
...

The knowledge bundle will include all files that both satisfy the whitelist regex and are specified in [replicationSettings:refineConf]. If multiple regex's are specified, the bundle will include the union of those files.

In this example, the knowledge bundle will include all files with extensions of either ".conf" or ".spec":

[replicationWhitelist]
allConf = *.conf
allSpec = *.spec

The names, such as allConf and allSpec, are used only for layering. That is, if you have both a global and a local copy of distsearch.conf, the local copy can be configured so that it overrides only one of the regex's. For instance, assume that the example shown above is the global copy and that you then specify a whitelist in your local copy like this:

[replicationWhitelist]
allConf = *.foo.conf

The two conf files will be layered, with the local copy taking precedence. Thus, the search head will distribute only files that satisfy these two regex's:

allConf = *.foo.conf
allSpec = *.spec

For more information on attribute layering in configuration files, see "Attribute precedence" in the Admin manual.

Caution: Replication whitelists are applied globally across all conf data, and are not limited to any particular app, regardless of where they are defined. Be careful to pull in only your intended files.

Edit the replicationSettings:refineConf stanza

The [replicationSettings:refineConf] stanza in distsearch.conf specifies the *.conf files and *.meta stanzas that get included in the replication bundle. If you want to modify the set of files in the bundle, add or delete them from this stanza.

The system default distsearch.conf file includes a version of this stanza that specifies the *.conf files that are normally included in the knowledge bundle:

[replicationSettings:refineConf]
# Replicate these specific *.conf files and their associated *.meta stanzas.
replicate.app               = true
replicate.authorize         = true
replicate.collections       = true
replicate.commands          = true
replicate.eventtypes        = true
replicate.fields            = true
replicate.segmenters        = true
replicate.literals          = true
replicate.lookups           = true
replicate.multikv           = true
replicate.props             = true
replicate.tags              = true
replicate.transforms        = true
replicate.transactiontypes  = true

If you want to replicate a .conf file that is not in the system default version of the [replicationSettings:refineConf] stanza, create a version of the stanza in $SPLUNK_HOME/etc/apps/<appname>/default/distsearch.conf and specify the *.conf file there. Similarly, you can remove files from the bundle by setting them to "false" in this stanza.

Limit the size of the knowledge bundle

You can also create a replication blacklist, using the [replicationBlacklist] stanza. This is most useful for limiting the size of the replication bundle, particularly in the case of very large files that do not need to be replicated to the search peers. The blacklist takes precedence over any whitelist.

Caution: Replication blacklists are applied globally across all conf data, and are not limited to any particular app, regardless of where they are defined. If you are defining an app-specific blacklist, be careful to constrain it to match only files that your application will not need.

Manage distributed server names

The name of each search head and search peer is determined by its serverName attribute, specified in server.conf. serverName defaults to the server's machine name.

In a distributed search cluster, all nodes must have unique names. The serverName has three specific uses:

  • For authenticating search heads. When search peers are authenticating a search head, they look for the search head's key file in /etc/auth/distServerKeys/<searchhead_name>/trusted.pem.
  • For identifying search peers in search queries. serverName is the value of the splunk_server field that you specify when you want to query a specific node. See "Retrieve events from indexes and distributed search peers" in the Search manual.
  • For identifying search peers in search results. serverName gets reported back in the splunk_server field.

Note: serverName is not used when adding search peers to a search head. In that case, you identify the search peers through their domain names or IP addresses.

The only reason to change serverName is if you have multiple instances of Splunk residing on a single machine, and they're participating in the same distributed search cluster. In that case, you'll need to change serverName to distinguish them.

Remove a search peer

Remove a search peer via Splunk Web

You can remove a search peer from a search head through the Distributed search page on Splunk Web. However, this only removes the search peer entry from the search head; it does not remove the search head key from the search peer. In most cases, this is not a problem and no further action is needed.

Remove a search peer via the CLI

Use the splunk remove search-server command to remove a search peer. When you run this command, make sure you specify the splunkd management port of the peer server. By default, this is 8089, although it might be different for your deployment.

You only need to provide credentials for the local machine. Use the -auth flag for your local credentials (in this example, for search peer 10.10.10.10):

splunk remove search-server -auth admin:password -url 10.10.10.10:8089

A message indicating success will be printed after peer is removed

Disable the trust relationship

If you need to disable the trust relationship between a search peer and a search head, you can delete the search-head-specific trusted.pem file from the directory $SPLUNK_HOME/etc/auth/distServerKeys/<searchhead_name>. It's unlikely that you'll need to do this.

Best practice: Forward search head data to the indexer layer

It is considered a best practice to forward all search head internal data to the search peer (indexer) layer. This has several advantages:

  • It accumulates all data in one place. This simplifies the process of managing your data: You only need to manage your indexes and data at one level, the indexer level.
  • It enables diagnostics for the search head if it goes down. The data leading up to the failure is accumulated on the indexers, where another search head can later access it.
  • By forwarding the results of summary index searches to the indexer level, all search heads have access to them. Otherwise, they're only available to the search head that generates them.

The preferred approach is to forward the data directly to the indexers, without indexing separately on the search head. You do this by configuring the search head as a forwarder. These are the main steps:

1. Make sure that all necessary indexes exist on the indexers. For example, the SoS app uses a scripted input that it puts data into a custom index. If you install SoS on the search head, you need to also install the SoS-TA add-on on the indexers, to provide the indexers with the necessary index settings for the data the app generates. On the other hand, since _audit and _internal exist on indexers as well as search heads, you do not need to create separate versions of those indexes to hold the corresponding search head data.

2. Configure the search head as a forwarder. Create an outputs.conf file on the search head that configures the search head for load-balanced forwarding across the set of search peers (indexers). For example:

[tcpout:my_search_peers]
server=10.10.10.1:9997,10.10.10.2:9997,10.10.10.3:9997

Note: Do not set indexAndForward=true in outputs.conf. If you do, the search head will both retain the data locally and forward it to the search peers.

For details on configuring outputs.conf, read "Configure forwarders with outputs.conf".

Synchronize system clocks across the distributed search environment

It is important that you synchronize the system clocks on all machines, virtual or physical, that are running Splunk Enterprise instances participating in distributed search. Specifically, this means your search heads and search peers. In the case of search head pooling or mounted bundles, this also includes the shared storage hardware. Otherwise, various issues can arise, such as bundle replication failures, search failures, or premature expiration of search artifacts.

The synchronization method you use depends on your specific set of machines. Consult the system documentation for the particular machines and operating systems on which you are running Splunk Enterprise. For most environments, Network Time Protocol (NTP) is the best approach to use.

PREVIOUS
Install a dedicated search head
  NEXT
Mount the knowledge bundle

This documentation applies to the following versions of Splunk® Enterprise: 5.0, 5.0.1, 5.0.2, 5.0.3, 5.0.4, 5.0.5, 5.0.6, 5.0.7, 5.0.8, 5.0.9, 5.0.10, 5.0.11, 5.0.12, 5.0.13, 5.0.14, 5.0.15, 5.0.16, 5.0.17, 5.0.18


Was this documentation topic helpful?

Enter your email address, and someone from the documentation team will respond to you:

Please provide your comments here. Ask a question or make a suggestion.

You must be logged into splunk.com in order to post comments. Log in now.

Please try to keep this discussion focused on the content covered in this documentation topic. If you have a more general question about Splunk functionality or are experiencing a difficulty with Splunk, consider posting a question to Splunkbase Answers.

0 out of 1000 Characters