Medium to large enterprise deployment: Search head cluster with multiple indexers

In the transition from a small enterprise to a medium enterprise deployment, you need to boost both indexing and search capacity. For indexing, you can continue to add indexers. For search, you can add search heads to service more users and more search activity.

The recommended approach for deploying multiple search heads is to combine the search heads in a search head cluster. Search head clusters allow users and searches to share resources across the set of search heads. They are also easier to manage than groups of individual search heads. Search head clusters require a minimum of three search heads.

The differences between a medium and a large enterprise deployment are mainly issues of scale and management. The fundamental deployment topologies are similar. They both employ a search head cluster with multiple indexers.

This topic describes how to implement a deployment consisting of:

• One search head cluster, containing multiple search heads
• Multiple indexers
• Multiple forwarders

Note: Instead of deploying multiple individual indexers, you can deploy an indexer cluster. This topology can be useful even if you do not want high data availability and the accompanying storage overhead. It still entails a small amount of overhead, but it offers the benefit of simplified indexer management. See "High availability deployment: Indexer cluster."

Medium enterprise deployment use case

A medium enterprise deployment provides greater horizontal scaling than a small enterprise deployment. It services larger numbers of users and searches. As your needs continue to increase, you can continue to add indexers and search heads.

Characteristics of this type of deployment include:

• Indexing volume between 100-300GB/day.
• Users numbering possibly a hundred or more.
• Up to a few thousand forwarders feeding load-balanced data to the indexers.

For details on the characteristics of a medium enterprise deployment, see "Types of deployments."

Large enterprise deployment use case

A large enterprise deployment provides even greater horizontal scaling.

Characteristics of this type of deployment include:

• Indexing volume ranging from 300GB to many TBs per day.
• A large number of users, potentially numbering in the several hundreds.
• Many thousands of forwarders.

For details on the characteristics of a large enterprise deployment, see "Types of deployments."

Architecture

This diagram shows a high-level view of the architecture for a medium or large enterprise deployment:

Starting from the bottom, the diagram illustrates three tiers of processing:

• Data input. Data enters the system through forwarders, which consume external data and forward the data to the indexers. You configure the forwarders to use their built-in load-balancing capabilities to spread the data across the set of indexers.

• Indexing. Indexers receive, index, and store incoming data from the forwarders.

• Search management. A search head cluster, consisting of three or more search head members, performs the search management function. The search heads in the cluster coordinate their activities to handle search requests, such as ad hoc requests from users and saved search requests, and to distribute the requests across the set of indexers. A deployer distributes apps to the members of the search head cluster.

Migration issues

This topic assumes that you are implementing a search head cluster deployment from scratch. If, instead, you are expanding from an existing, non-clustered deployment, the process is similar but there are a few additional issues to be aware of:

• You must use a new Splunk Enterprise instance for each search head in a cluster. You can migrate your current search head's settings and apps to a search head cluster, but you cannot reuse the search head itself. See "Migrate from a standalone search head to a search head cluster" in the Distributed Search manual.
• You must configure your licensing to accommodate the additional instances.

The implementation framework procedure covers these issues.

Implementation framework

To implement a search head cluster with indexers and forwarders:

1. Install and deploy the search head cluster. See "Deploy a search head cluster" in the Distributed Search manual.

2. (Migration only) If you are migrating from a smaller deployment consisting of a standalone search head, you can migrate the settings from the old search head. See "Migrate from a standalone search head to a search head cluster" in the Distributed Search manual.

3. Install the instances that you plan to use as indexers. For installation instructions, see "Installation overview" in the Installation Manual.

(Migration only) If you are migrating from a smaller deployment, you can continue to use the existing indexers. You can also add new indexer instances, as needed.

4. Connect the search heads to the indexers, also known as search peers. See "Connect the search heads in clusters to search peers" in the Distributed Search manual.

5. Configure the Splunk Enterprise license. If you have an existing license, you must ensure that it covers any new instances. See "How Splunk Enterprise licensing works" in the Admin Manual.

6. On each new indexer, configure the receiving port. The forwarders send data to the indexer through this port. See "Enable a receiver" in the Forwarding Data manual.

7. Install universal forwarders on the machines hosting your data sources, and configure the forwarders to send data to the set of indexers. For information on installing and configuring universal forwarders, see Install the universal forwarder software in the Universal Forwarder manual.

It is recommended that you configure the forwarders to load-balance their data across the set of indexers. See "Set up load balancing" in the Forwarding Data manual.

(Migration only) If you already have forwarders from a previous deployment, and you added indexers in step 3, reconfigure the existing forwarders to load-balance across the entire set of indexers, including the new indexers.

8. Configure the inputs to the forwarders, so that data begins to enter the system. See "Configure your inputs" in the Getting Data In manual.

Next steps

Once you have your Splunk Enterprise instances up and running and talking to each other, you can further refine your system and prepare the data and its presentation for the benefit of your end users. For a summary of the types of activities you need to perform now, see "Post-deployment activities."

To scale further

You can continue to scale your deployment as needed. To increase your indexing and search capacity, the first step is to add more indexers. To do this, install another Splunk Enterprise instance and configure the search heads and forwarders to connect with it.

To service more users and continue to increase search capacity beyond a certain level, you must add another search head to the cluster. See "Add a cluster member" in the Distributed Search manual.

For detailed information on determining when to add more instances, and whether to add only indexers or both indexers and search heads, see:

• "Types of deployments" in this manual.

• Summary of performance recommendations" in the Capacity Planning manual.