Small enterprise deployment: Single search head with multiple indexers

To increase indexing and search capacity beyond a certain point, you must transition from a single Splunk Enterprise instance to a multi-instance deployment. You split the search management and indexing functions, allocating them to separate instances running on separate machines. At this first level of scaling, you typically deploy a single search head that communicates with two or three indexers. This type of deployment is characteristic of a multi-departmental, or small enterprise, solution.

This topic describes how to implement a deployment consisting of:

• One search head
• Multiple indexers
• Multiple forwarders

Note: Instead of deploying multiple individual indexers, you can deploy an indexer cluster. This topology can be useful even if you do not want high data availability and the accompanying storage overhead. It still entails a small amount of overhead, but it offers the benefit of simplified indexer management. See High availability deployment: Indexer cluster.

Use case

This distributed search scenario provides the first level of horizontal scaling. It allows users to run searches across a set of indexers. As your needs increase further, you can add more indexers.

Characteristics of this type of deployment include:

• Indexing volume between 20 and 100GB/day.
• Between 10 and 100 users.
• Up to several hundred forwarders feeding data to the indexers. The forwarders typically make use of load balancing to distribute the data across the set of indexers.

For details on the characteristics of a small enterprise deployment, see Types of deployments.

Architecture

This diagram shows a high-level view of the architecture for this type of deployment:

Starting from the bottom, the diagram illustrates three tiers of processing:

• Data input. Data enters the system through forwarders, which consume external data, perform a small amount of preprocessing on it, and then forward the data to the indexers. You usually configure the forwarders to use their built-in load-balancing capabilities to spread the data across the set of indexers.

• Indexing. Two or three indexers receive, index, and store incoming data from a set of forwarders. The indexers also search that data, in response to requests from the search head.

• Search management. A single search head performs the search management function. It handles search requests, such as ad hoc requests from users and saved search requests, and distributes the requests across the set of indexers, which perform the actual searches on their local data. The search head then consolidates the results from the indexers and serves them to the users. The search head provides the user with various tools, such as dashboards, to assist the search experience.

Migration issues

This topic assumes that you are implementing this deployment from scratch. If, instead, you are expanding an existing single-instance deployment, the process is similar but there are a few additional issues to be aware of:

• Make your current single-instance deployment an indexer, not a search head, in the new deployment. This instance is likely to be provisioned more appropriately for the needs of an indexer. The data already on the instance will continue to be searchable in the new deployment.
• Transfer the standalone instance's apps and knowledge objects to the new search head instance.
• Configure your licensing to accommodate the additional instances.

The implementation framework procedure includes steps to handle these issues.

Implementation framework

To implement this type of scenario:

1. Install Splunk Enterprise instances for the indexers and search head. For example, to deploy a single search head with two indexers, install three instances. For installation instructions, see Installation overview in the Installation Manual.

Note: Search heads and indexers have different hardware requirements. For information on provisioning the hardware for your instances, see Reference hardware in the Capacity Planning manual.

2. (Migration only) If you are migrating from a standalone instance, use that instance as one of the indexers. The data already on the instance continues to be available in the new environment.

3. (Migration only) If you are migrating from a standalone instance, transfer the existing set of knowledge objects and apps to the new search head instance:

a. Update the former standalone instance, if necessary, so that it is running the same version as the new search head instance.

b. Copy the contents of the standalone instance's $SPLUNK_HOME/etc/apps and $SPLUNK_HOME/etc/users directories to the same locations on the search head.

c. Restart the search head.

4. On the instance that you have chosen as your search head, configure the indexer instances as search peers to the instance. This step formally declares the respective roles (indexers, search head) for the instances. See Add search peers to the search head in the Distributed Search manual.

5. Configure the Splunk Enterprise license. If you have an existing license, ensure that it covers the new instances. See How Splunk Enterprise licensing works in the Admin Manual.

6. On each indexer, configure the receiving port. The forwarders send data to the indexer through this port. See Enable a receiver in the Splunk Universal Forwarder manual.

7. Install universal forwarders on the machines hosting your data sources, and configure the forwarders to send data to the set of indexers. How you do this depends on your needs and preferences, as well as on how many forwarders you are deploying. For example:

• You can install forwarders one-by-one or simultaneously across many machines.
• You can install forwarders and then configure them later, or install and configure forwarders at the same time.
• You can configure forwarders either manually or by means of the deployment server or third-party software.

For information on installing and configuring universal forwarders, see Deploy the universal forwarder in the Universal Forwarder' manual.

It is recommended that you configure the forwarders to load-balance their data across the set of indexers, rather than sending data to just a single indexer. See Set up load balancing in the Forwarding Data manual.

(Migration only) If you already have some forwarders from a previous deployment, reconfigure them to load-balance across all the indexers.

8. Configure the inputs to the forwarders, so that data begins to enter the system. See Configure your inputs in the Getting Data In manual.

Next steps

Once you have your Splunk Enterprise instances up and running and talking to each other, you can further refine your system and prepare the data and its presentation for the benefit of your end users. For a summary of the types of activities you need to perform now, see Post-deployment activities.

To scale further

To increase your indexing and search capacity, the first step is to add more indexers. To do this, install another Splunk Enterprise instance, and configure the search head to treat it as a search peer.

To service more users and increase search capacity beyond a certain level, you must eventually add more search heads. Ordinarily, the best way to deploy multiple search heads is to deploy a search head cluster. See Medium to large enterprise deployment: Search head cluster with multiple indexers.

For detailed information on determining when to add more instances, and whether to add only indexers or both indexers and search heads, see:

• Types of deployments in this manual.

• Summary of performance recommendations in the Capacity Planning manual.