Configure the peer nodes
Important: Before reading this topic, you should understand how configuration files work in Splunk. Read "About configuration files" and the topics that follow it in the Admin Manual.
Most peer configuration happens during initial deployment of the cluster:
1. When you enable the peer, you specify its master, as well as the port on which it receives replicated data. See "Enable the peer nodes".
2. After you enable the set of peers, you configure their indexes. See "Configure the peer indexes".
3. Finally, you configure their inputs, usually by means of forwarders. See "Use forwarders to get your data".
These are the key steps in configuring a peer. However, you might also need to update the initial peer configurations later, as with any indexer. For example, to configure event processing, you might need to edit
transforms.conf. This topic tells you how to update a peer's configuration post-deployment.
As a best practice, you should treat your peers as interchangeable, and therefore you should maintain identical versions of most configuration files across all peers. For the cluster to properly replicate data and handle node failover, peers must share the same indexing functionality, and they cannot do this if certain key files vary from peer to peer. In particular, the set of index stanzas in
indexes.conf should ordinarily be identical across all peers, aside from very limited exceptions described later. In addition, it's important that index-time processing be the same across the peers.
This topic describes how to maintain identical configurations across the set of peers. It also describes how to set configurations on a peer-by-peer basis when necessary.
Manage common configurations across all peers
As far as possible, you should use a common set of configuration files across all peers in a cluster. In particular, the
indexes.conf file should ordinarily be identical across all the peer nodes, because all peers must share the same set of clustered indexes. It is also a good idea to maintain the same
transforms.conf files across all peers. Beyond these three key files, you can greatly simplify cluster management by maintaining identical versions of most other configuration files, such as
inputs.conf, across all peers.
Important: There are a few crucial differences in how you manage common peer configuration files compared to configurations for standalone indexers:
- Do not make configuration changes on individual peers that will modify configurations you need to maintain on a cluster-wide basis. For example, do not use Manager or the CLI to configure index settings. Also, do not edit those configuration files directly on the peers. Instead, edit the configuration files in a central location from where they can be distributed to all peers, using the configuration bundle method described later in this topic.This provides the only way to ensure that all peers use the same versions of these files.
- Do not use deployment server to manage common configuration files, like
indexes.conf, across peer nodes. Instead, use the configuration bundle method introduced in this topic and described in detail in "Update common peer configurations". This update method ensures that all the peers share the same versions of those files.
- On a standalone Splunk instance, there can be multiple versions of most configuration files residing in numerous locations, as described in "About configuration files" in the Admin manual. Splunk layers the versions according to the precedence rules discussed in the topic "Configuration file precedence". However, for configurations that you want to be identical across all peers, you must consolidate all non-default layers; otherwise, multiple file versions can subvert your attempt to create a single cross-peer configuration. When updating a particular configuration file across a set of peers, first consolidate all non-default versions of that file into a single file that you can then distribute across all the peers.
Configure indexes.conf for all peers
For the purposes of replicating data, it's critical that the set of clustered indexes in the
indexes.conf file be identical across all peers in a cluster. For information on how to configure this file, read the topic "Configure the peer indexes".
Note: Under limited circumstances (for example, to perform local testing or monitoring), you might want to add an index (or an app with a new index) to one peer but not the others. You can do this by creating a peer-specific
indexes.conf, so long as you're careful about how you configure the index and are clear about the ramifications. The data in such an index will not get replicated. The peer-specific
indexes.conf supplements, but does not replace, the common version of the file that all peers get. See "Add an index to a single peer", later in this topic, for details.
How to distribute updated configurations to all the peers
To distribute new or edited configuration files across all the peers, you first place them in a special location on the master. You then tell the master to distribute the files to the peer nodes. The set of configuration files common to all peers, which is managed from the master and distributed to the peers in a single operation, is known as the configuration bundle.
Your use of this distribution method is not limited to
transforms.conf. You can use the same method to distribute any identical configuration files to all peers. For example, if all your peers are able to share a single set of inputs, you can use this method to distribute a common
inputs.conf file to all peers.
For details of how to edit and update configurations across all the peers, read the topic "Update common peer configurations".
Note: Although it's possible to distribute common peer configuration files using some different distribution method, it's not recommended. When you distribute via the master node, as described in "Update common peer configurations", the master orchestrates the distribution to ensure that all peers use the same set of configuration files, including the same set of clustered indexes. If you use another distribution method, you must particularly ensure that settings for any new clustered indexes are successfully distributed to all peers, and that all the peers have been restarted, before you start sending data to the new indexes.
Manage configurations on a peer-by-peer basis
You might need to handle some configurations on a peer-by-peer basis. For most purposes, however, it's better to use the same configurations across all peers, so that the peers are interchangeable.
Configure data inputs
It's recommended that you use forwarders to handle data inputs to peers. For information on configuring this process, read "Use forwarders to get your data".
If you want to input data directly to a peer, without a forwarder, you can configure your inputs on the peer in the same way as for any indexer. For more information, read "Configure your inputs" in the Getting Data In Manual.
No matter what method you use to configure your inputs, all configurations get written to an inputs.conf file.
Important: Although you can configure inputs on a peer-by-peer basis, consider whether your needs would allow you to use a single set of inputs across all peers. This should be possible if all data is being channeled through forwarders, and the receiving ports on all peers can be the same. If so, you can use the master node to manage a common
inputs.conf file, as described in "Update common peer configurations".
Add an index to a single peer
If you need to add an index to a single peer, you can do so by creating a separate
indexes.conf file on the peer. However, the data in the new index will remain only on that peer and will not get replicated. The main use case for this is to perform some sort of local testing or monitoring, possibly involving an app that you download to only that one peer. The peer-specific
indexes.conf supplements, but does not replace, the common version of the file that all peers get.
If you create a new version of
indexes.conf for a single peer, you can put it in any of the acceptable locations for an indexer, as outlined in "About configuration files" in the Admin Manual. The one place where you cannot put the file is under
$SPLUNK_HOME/etc/slave-apps, which is where the configuration bundle resides on the peer. If you put it there, it will get overwritten the next time the peer downloads a configuration bundle.
indexes.conf file gets added as part of an app that you've downloaded to a single peer, it will go into the usual location for downloaded apps, under
Important: If you add a local index, leave its
repFactor attribute set to the default value of 0. Do not set it to
auto. If you set it to
auto, the index's data will be replicated to other peers in the cluster, Since the other peers will not have a configuration stanza for the new index, there will be nowhere on those peers to store the replicated data, resulting in various, potentially serious, problems.
Make other configuration changes
If you need to make some configuration changes specific to individual peers, you can configure those peers in the same way as any other Splunk instance. You can use Manager or the CLI, or you can directly edit the configuration files.
Restart the peer
As with any indexer, you usually need to restart a peer after you change its configuration. Unlike non-clustered indexers, however, you should not use the CLI
splunk restart command to restart the peer. Instead, use the restart capability in Splunk Manager. For detailed information on how to restart a cluster peer, read "Restart a cluster peer".
Configure the master
Configure the peer indexes
This documentation applies to the following versions of Splunk® Enterprise: 5.0, 5.0.1