High availability deployment: Indexer cluster

To ensure high availability for your data, you can deploy an indexer cluster. Indexer clusters are groups of indexers configured to replicate each others' data, so that the system keeps multiple copies of all data. This process is known as index replication. Indexer clusters prevent data loss while promoting data availability for searching. Indexer clusters can also be simpler to manage than groups of individual indexers.

The trade-off with indexer clusters is that you need additional storage, to handle the replicated copies. You can control the degree of index replication, and thus the storage requirements, to match the availability needs of your enterprise.

For an introduction to indexer clustering, along with more details on the benefits and trade-offs, see "About indexer clusters and index replication" in the Managing Indexers and Clusters of Indexers manual.

You can use indexer clustering with an enterprise deployment of any size.

As your search needs grow, you can combine an indexer cluster with a search head cluster.

This topic describes how to implement a deployment consisting of:

• One or more individual search heads, or a search head cluster
• One indexer cluster
• Multiple forwarders

High availability use case

The main use case for an indexer cluster is an enterprise deployment that requires high data availability and is willing to allocate the additional disk space necessary to store multiple copies of the data.

Simplified management use case

You can also implement an indexer cluster without replication. When you eliminate replication, you lose some of the key benefits of an indexer cluster, such as data availability and data recovery, but you still gain the benefits of simplified management of multiple indexers. See "Use indexer clusters to scale indexing" in the Managing Indexers and Clusters of Indexers manual.

Architecture

The main types of architecture for an indexer cluster are:

• Indexer cluster with one or more individual search heads
• Indexer cluster with search head cluster

Indexer cluster with individual search heads

This diagram shows a high-level view of the architecture for an indexer cluster with one or more individual search heads:

Starting from the bottom, the diagram illustrates three tiers of processing:

• Data input. Data enters the system through forwarders, which consume external data and forward the data to the indexers. You configure the forwarders to use their built-in load-balancing capabilities to spread the data across the set of indexers.

• Indexing. Indexers, or cluster peer nodes, receive, index, and store incoming data from the forwarders.

• Search management. One or more individual search heads perform the search management function.

A manager (master) node regulates the functioning of the cluster nodes, but does not index, store, or search data.

Indexer cluster with search head cluster

You can combine an indexer cluster with a search head cluster. This is the recommended approach if you need to deploy multiple search heads:

The main difference compared to the previous architecture is that a search head cluster replaces the individual search heads. The indexer cluster interacts with the search head cluster in essentially the same way that it would interact with individual search heads.

Migration issues

This topic assumes that you are implementing an indexer cluster deployment from scratch. If, instead, you are expanding from a set of non-clustered indexers, you can incorporate the existing indexers into your cluster, but there are a few issues to be aware of:

• Data already on the individual indexers remains available for searching, but it does not get replicated.
• You can no longer use the deployment server to manage your apps. Instead, you must use the configuration bundle method built into the indexer cluster. This requires that you migrate your apps to the cluster.

For details on these and other issues, see:

• "Key differences between clustered and non-clustered deployments of indexers" in the Managing Indexers and Clusters of Indexers manual.
• "Migrate non-clustered indexers to a clustered environment" in the Managing Indexers and Clusters of Indexers manual.

The implementation framework procedure mentions migration issues briefly.

Implementation framework

To implement an indexer cluster:

1. Install the necessary Splunk Enterprise instances, and deploy the indexer cluster. See "Indexer cluster deployment overview" in the Managing Indexers and Clusters of Indexers manual.

2. (Migration only) If you are migrating from a deployment consisting of one or more standalone indexers, also read "Migrate non-clustered indexers to a clustered environment" in the Managing Indexers and Clusters of Indexers manual.

3. If you want to implement a search head cluster for the search tier, see "Deploy a search head cluster" and "Integrate the search head cluster with an indexer cluster" in the Distributed Search manual.

4. Configure the Splunk Enterprise license. If you have an existing license, you must ensure that it covers any new instances. See "How Splunk Enterprise licensing works" in the Admin Manual. Also, see the section on licensing information in "System requirements and other deployment considerations for indexer clusters" in the Managing Indexers and Clusters of Indexers manual.

5. Install universal forwarders on the machines hosting your data sources, if you have not already done so. See Install the universal forwarder software in the Universal Forwarder manual.

6. Connect the peer nodes to the forwarders:

a. Decide on the method to use. See "Use forwarders to get data into the indexer cluster" in the Managing Indexers and Clusters of Indexers manual.

b. Depending on your decision, follow either "Use indexer discovery to connect forwarders to peer nodes" or "Connect forwarders directly to peer nodes" in the Managing Indexers and Clusters of Indexers manual.

7. Configure the inputs to the forwarders, so that data begins to enter the system. See "Configure your inputs" in the Getting Data In manual.

Next steps

Once you have your Splunk Enterprise instances up and running and talking to each other, you can further refine your system and prepare the data and its presentation for the benefit of your end users. For a summary of the types of activities you need to perform now, see "Post-deployment activities."

To scale further

You can scale the cluster as needed. To increase indexing and search capacity, the first step is to add another indexer. To do this, enable a new Splunk Enterprise instance as a peer node.

To service more users and continue to increase search capacity beyond a certain level, you must add more search heads. The recommended way to include multiple search heads in an indexer cluster is to deploy them in a search head cluster. See "Migrate from a standalone search head to a search head cluster" in the Distributed Search manual.

To add more search heads to an existing search head cluster, see "Add a cluster member" in the Distributed Search manual.

For detailed information on determining when to add more instances, and whether to add only indexers or both indexers and search heads, see:

• "Types of deployments" in this manual.

• Summary of performance recommendations" in the Capacity Planning manual.

If your enterprise spans multiple sites, you can implement a multisite indexer cluster instead of a single-site cluster. See "Multisite indexer cluster deployment overview" in the Managing Indexers and Clusters of Indexers manual. If you already have a single-site cluster and want to convert it to a multisite cluster, see "Migrate an indexer cluster from single-site to multisite" in the Managing Indexers and Clusters of Indexers manual.