Splunk® Enterprise

Managing Indexers and Clusters of Indexers

Download manual as PDF

Splunk Enterprise version 5.0 reached its End of Life on December 1, 2017. Please see the migration information.
This documentation does not apply to the most recent version of Splunk. Click here for the latest version.
Download topic as PDF

Indexes, indexers, and clusters

This manual discusses Splunk Enterprise data repositories and the Splunk Enterprise components that create and manage them.

The index is the repository for Splunk Enterprise data. Splunk Enterprise transforms incoming data into events, which it stores in indexes.

An indexer is a Splunk Enterprise instance that indexes data. For small deployments, a single instance might perform other Splunk Enterprise functions as well, such as data input and search management. In a larger, distributed deployment, however, the functions of data input and search management are allocated to other Splunk Enterprise components. This manual focuses exclusively on the indexing function, in the context of either a single-instance or a distributed deployment.

A cluster is a group of indexers configured to replicate each others' data, so that the system keeps multiple copies of all data. This process is known as index replication. By maintaining multiple, identical copies of data, clusters prevent data loss while promoting data availability for searching.


As Splunk Enterprise indexes your data, it creates a bunch of files. These files fall into two categories:

  • The raw data in compressed form (rawdata)
  • Indexes that point to the raw data, plus some metadata files (index files)

Together, these files constitute the Splunk Enterprise index. The files reside in sets of directories organized by age. These directories are called buckets. For more information, see "How Splunk Enterprise stores indexes" in this manual.

Splunk Enterprise manages its indexes to facilitate flexible searching and fast data retrieval, eventually archiving them according to a user-configurable schedule. Splunk Enterprise handles everything with flat files; it doesn't require any third-party database software running in the background.

During indexing, Splunk Enterprise processes incoming data to enable fast search and analysis, storing the results in the index as events. While indexing, Splunk Enterprise enhances the data in various ways, including by:

  • Separating the datastream into individual, searchable events.
  • Creating or identifying timestamps.
  • Extracting fields such as host, source, and sourcetype.
  • Performing user-defined actions on the incoming data, such as identifying custom fields, masking sensitive data, writing new or modified keys, applying breaking rules for multi-line events, filtering unwanted events, and routing events to specified indexes or servers.

This indexing process is also known as event processing.

To start indexing, you simply specify the data inputs that you want Splunk Enterprise to index. You can add more inputs at any time, and Splunk Enterprise will begin indexing them as well. See "What Splunk Enterprise can index" in the Getting Data In Manual to learn how to add data inputs. The Getting Data In Manual also describes how you can configure index-time event processing to meet the needs of your data. See "Overview of event processing".

Splunk Enterprise, by default, puts all user data into a single, preconfigured index. It also employs several other indexes for internal purposes. You can add new indexes and manage existing ones to meet your data requirements. See "Manage indexes" in this manual.


The indexer is the Splunk Enterprise component that creates and manages indexes. The primary functions of an indexer are:

  • Indexing incoming data.
  • Searching the indexed data.

In single-machine deployments consisting of just one Splunk Enterprise instance, the indexer also handles the data input and search management functions. This type of small deployment might handle the needs of a single department in an organization.

For larger-scale needs, indexing is split out from the data input function and sometimes from the search management function as well. In these larger, distributed deployments, the Splunk Enterprise indexer might reside on its own machine and handle only indexing, along with searching of its indexed data. In those cases, other Splunk Enterprise components take over the non-indexing roles. Forwarders consume the data, indexers index and search the data, and search heads coordinate searches across the set of indexers. Here's an example of a scaled-out deployment:

Horizontal scaling new2.png

For more information on using indexers in a distributed deployment, see "Indexers in a distributed deployment" in this manual.

A Splunk indexer is simply a Splunk Enterprise instance. To learn how to install a Splunk Enterprise instance, read the Installation Manual.


A cluster is a group of Splunk Enterprise nodes that, working in concert, provide a redundant indexing and searching capability. There are three types of nodes in a cluster:

  • A single master node to manage the cluster. The master node is a specialized type of indexer.
  • Several peer nodes that handle the indexing function for the cluster, indexing and maintaining multiple copies of the data and running searches across the data.
  • One or more search heads to coordinate searches across all the peer nodes.

Splunk Enterprise clusters feature automatic failover from one peer node to the next. This means that, if one or more peers fail, incoming data continues to get indexed and indexed data continues to be searchable.

The information in the first half of this manual on indexing is relevant for all indexers, independent of whether they're part of a cluster. However, clusters require additional configuration, beyond what's needed for a stand-alone indexer. For more information on clusters, read the second half of this manual, starting with "About clusters and index replication".

How indexing works

This documentation applies to the following versions of Splunk® Enterprise: 5.0, 5.0.1, 5.0.2, 5.0.3, 5.0.4, 5.0.5, 5.0.6, 5.0.7, 5.0.8, 5.0.9, 5.0.10, 5.0.11, 5.0.12, 5.0.13, 5.0.14, 5.0.15, 5.0.16, 5.0.17, 5.0.18

Was this documentation topic helpful?

Enter your email address, and someone from the documentation team will respond to you:

Please provide your comments here. Ask a question or make a suggestion.

You must be logged into splunk.com in order to post comments. Log in now.

Please try to keep this discussion focused on the content covered in this documentation topic. If you have a more general question about Splunk functionality or are experiencing a difficulty with Splunk, consider posting a question to Splunkbase Answers.

0 out of 1000 Characters