Use clusters to scale indexing
The main purpose of clusters is to enable index replication. However, clusters can also be generally useful in scale-out deployment topologies as a way to manage multiple indexers, even when index replication isn't a requirement.
For example, say you want to create a deployment of three indexers and one search head, so that you can index larger quantities of data than a single indexer is capable of. The customary way of doing this, and the only way possible prior to Splunk 5.0, is to set up each of the indexers independently, add in a search head, and then use a tool like deployment server to coordinate the indexer configurations.
With clustering, you can instead configure this deployment scenario as a cluster, with three peer nodes replacing the three independent indexers. Even if you don't need index replication and its key advantages like data availability and disaster tolerance, there are several reasons why it might be beneficial to use a cluster to coordinate multiple indexer instances:
- Simplified management and coordination of indexer configuration (in place of using deployment server or performing manual updates). See "Update common peer configurations" for details.
- Simplified set up and control of distributed search. See "Enable the search head".
- Better insight into the state of your indexers through the clustering dashboards. See "View the master dashboard".
- Ability to take advantage of additional cluster management capabilities as they're developed.
The main downsides of employing a cluster for scaling indexing capacity are these:
- You must install an additional Splunk instance to function as the cluster master node.
- All cluster components must be on the same high-speed network.
- The cluster does not support heterogeneous indexers. All peer nodes in a cluster must use the same
indexes.confconfiguration. For further details, see the next section, "Cluster peer management compared to deployment server".
- You cannot use the deployment server to distribute configurations or apps across the cluster peers. For further details, see the next section, "Cluster peer management compared to deployment server".
Cluster peer management compared to deployment server
One useful cluster feature is the ability to manage and update the configuration for all indexers (peer nodes) from a central location, the master node. In that respect, it's similar in function to the deployment server. Unlike the deployment server, however, peer management does not have any concept of server classes. Because of this, and because of the way clusters coordinate their activities, you cannot specify different app or
indexes.conf configurations for different groups of indexers. (All peer nodes in a cluster must use the same
indexes.conf configurations, as well as some other configurations, as described in "Configure the peer nodes".) If you need to maintain a heterogeneous set of indexers, you cannot employ clusters for scaling purposes.
In addition, deployment server or third party distributed configuration management software such as Puppet or Chef are unsupported methods to distribute configurations or apps across the peer nodes.
On the other hand, the configuration bundle method used to download updates to peer nodes has certain advantages over the deployment server. Specifically, it not only distributes updates, it also validates them on the peers, and then initiates a rolling restart of the peers. See "Update common peer configurations" for details.
Configure a cluster for scale-out deployment
To set up a cluster for scale-out deployment, without index replication, just set both the replication factor and search factor to 1. This causes the cluster to function purely as a coordinated set of Splunk instances, without data replication. The cluster will not make any duplicate copies of the data, so you can keep storage size and processing overhead to a minimum.
Use forwarders to get your data
Migrate non-clustered indexers to a clustered environment
This documentation applies to the following versions of Splunk® Enterprise: 5.0, 5.0.1, 5.0.2, 5.0.3, 5.0.4, 5.0.5, 5.0.6, 5.0.7, 5.0.8, 5.0.9, 5.0.10, 5.0.11, 5.0.12, 5.0.13, 5.0.14, 5.0.15, 5.0.16, 5.0.17, 5.0.18