Splunk® Enterprise

Managing Indexers and Clusters of Indexers

Download manual as PDF

Splunk Enterprise version 5.0 reached its End of Life on December 1, 2017. Please see the migration information.
This documentation does not apply to the most recent version of Splunk. Click here for the latest version.
Download topic as PDF

Indexers in a distributed deployment

The indexer is the Splunk component that creates and manages indexes. The primary functions of an indexer are:

  • Indexing incoming data.
  • Searching the indexed data.

In single-machine deployments consisting of just one Splunk instance, the indexer also handles the data input and search management functions.

For larger-scale needs, indexing is split out from the data input function and sometimes from the search management function as well. In these larger, distributed deployments, the Splunk indexer might reside on its own machine and handle only indexing, along with searching of its indexed data. In those cases, other Splunk components take over the non-indexing roles.

For instance, you might have a set of Windows and Linux machines generating events, which need to go to a central Splunk indexer for consolidation. Usually the best way to do this is to install a lightweight instance of Splunk, known as a forwarder, on each of the event-generating machines. These forwarders handle data input and send the data across the network to the Splunk indexer residing on its own machine.

Similarly, in cases where you have a large amount of indexed data and numerous concurrent users searching on it, it can make sense to split off the search management function from indexing. In this type of scenario, known as distributed search, one or more search heads distribute search requests across multiple indexers. The indexers still perform the actual searching of their own indexes, but the search heads manage the overall search process across all the indexers and present the consolidated search results to the user.

Here's an example of a scaled-out deployment:

Horizontal scaling new3.png

While the fundamental issues of indexing and event processing remain the same for distributed deployments, it is important to take into account deployment needs when planning your indexing strategy.

Note: Although a cluster can be an integral part of a larger, distributed deployment, the cluster components themselves must reside together on a high speed network. For information about clusters, see "About clusters and index replication".

Forward data to an indexer

To forward remote data to an indexer, you use forwarders, which are Splunk instances that receive data inputs and then consolidate and send the data to a Splunk indexer. Forwarders come in two flavors:

  • Universal forwarders. These maintain a small footprint on their host machine. They perform minimal processing on the incoming data streams before forwarding them on to an indexer, also known as the receiver.
  • Heavy forwarders. These retain most of the functionality of a full Splunk instance. They can parse data before forwarding it to the receiving indexer. (See "How indexing works" for the distinction between parsing and indexing.) They can store indexed data locally and also forward the parsed data to a receiver for final indexing on that machine as well.

Both types of forwarders tag data with metadata such as host, source, and source type, before forwarding it on to the indexer.

Forwarders allow you to use resources efficiently when processing large quantities or disparate types of data coming from remote sources. They also enable a number of interesting deployment topologies, by offering capabilities for load balancing, data filtering, and routing.

For an extended discussion of forwarders, including configuration and detailed use cases, see "About forwarding and receiving" in the Distributed Deployment manual.

Search across multiple indexers

In distributed search, Splunk search heads send search requests to Splunk indexers and then merge the results back to the user. This is useful for a number of purposes, including horizontal scaling, access control, and managing geo-dispersed data.

For an extended discussion of distributed search, including configuration and detailed use cases, see "About distributed search" in the Distributed Deployment manual.

Clusters of indexers also use search heads to coordinate searches across the cluster's peer nodes. See "About clusters and index replication".

PREVIOUS
Install an indexer
  NEXT
About managing indexes

This documentation applies to the following versions of Splunk® Enterprise: 5.0, 5.0.1, 5.0.2, 5.0.3, 5.0.4, 5.0.5, 5.0.6, 5.0.7, 5.0.8, 5.0.9, 5.0.10, 5.0.11, 5.0.12, 5.0.13, 5.0.14, 5.0.15, 5.0.16, 5.0.17, 5.0.18


Was this documentation topic helpful?

Enter your email address, and someone from the documentation team will respond to you:

Please provide your comments here. Ask a question or make a suggestion.

You must be logged into splunk.com in order to post comments. Log in now.

Please try to keep this discussion focused on the content covered in this documentation topic. If you have a more general question about Splunk functionality or are experiencing a difficulty with Splunk, consider posting a question to Splunkbase Answers.

0 out of 1000 Characters