How indexing works
Event processing and the data pipeline
Data enters the indexer and proceeds through a pipeline where event processing occurs. Finally, the processed data it is written to disk. This pipeline consists of several shorter pipelines that are strung together. A single instance of this end-to-end data pipeline is called a pipeline set.
Event processing occurs in two main stages, parsing and indexing. All data that comes into Splunk Enterprise enters through the parsing pipeline as large (10,000 bytes) chunks. During parsing, Splunk Enterprise breaks these chunks into events which it hands off to the indexing pipeline, where final processing occurs.
While parsing, Splunk Enterprise performs a number of actions, including:
- Extracting a set of default fields for each event, including
- Configuring character set encoding.
- Identifying line termination using linebreaking rules. While many events are short and only take up a line or two, others can be long.
- Identifying timestamps or creating them if they don't exist. At the same time that it processes timestamps, Splunk identifies event boundaries.
- Splunk can be set up to mask sensitive event data (such as credit card or social security numbers) at this stage. It can also be configured to apply custom metadata to incoming events.
In the indexing pipeline, Splunk Enterprise performs additional processing, including:
- Breaking all events into segments that can then be searched upon. You can determine the level of segmentation, which affects indexing and searching speed, search capability, and efficiency of disk compression.
- Building the index data structures.
- Writing the raw data and index files to disk, where post-indexing compression occurs.
The breakdown between the parsing and indexing pipelines is of relevance mainly when deploying forwarders. Heavy forwarders can run raw data through the parsing pipeline and then forward the parsed data on to indexers for final indexing. Universal forwarders do not parse data in this way. Instead, universal forwarders forward the raw data to the indexer, which then processes it through both pipelines. Note, however, that both types of forwarders do perform a type of parsing on certain structured data. See Extract data from files with headers in Getting Data In.
For more information about events and how the indexer transforms data into events, see the chapter Configure event processing in Getting Data In.
Note: Indexing is an I/O-intensive process.
This diagram shows the main processes inherent in indexing:
Note: This diagram represents a simplified view of the indexing architecture. It provides a functional view of the architecture and does not fully describe Splunk Enterprise internals. In particular, the parsing pipeline actually consists of three pipelines: parsing, merging, and typing, which together handle the parsing function. The distinction can matter during troubleshooting, but does not generally affect how you configure or deploy Splunk Enterprise.
For a more detailed discussion of the data pipeline and how it affects deployment decisions, see How data moves through Splunk Enterprise: the data pipeline in Distributed Deployment.
What's in an index?
Splunk Enterprise stores all of the data it processes in indexes. An index is a collection of databases, which are subdirectories located in
$SPLUNK_HOME/var/lib/splunk. Indexes consist of two types of files: rawdata files and index files. See How Splunk Enterprise stores indexes.
Default set of indexes
Splunk Enterprise comes with a number of preconfigured indexes, including:
- main: This is the default Splunk Enterprise index. All processed data is stored here unless otherwise specified.
- _internal: Stores Splunk Enterprise internal logs and processing metrics.
- _audit: Contains events related to the file system change monitor, auditing, and all user search history.
A Splunk Enterprise administrator can create new indexes, edit index properties, remove unwanted indexes, and relocate existing indexes. Splunk Enterprise administrators manage indexes through Splunk Web, the CLI, and configuration files such as
indexes.conf. See Managing indexes.
Indexes, indexers, and indexer clusters
Index time versus search time
This documentation applies to the following versions of Splunk® Enterprise: 6.3.0, 6.3.1, 6.3.2, 6.3.3, 6.3.4, 6.3.5, 6.3.6, 6.3.7, 6.3.8, 6.3.9, 6.3.10, 6.3.11, 6.3.12, 6.3.13, 6.3.14, 6.4.0, 6.4.1, 6.4.2, 6.4.3, 6.4.4, 6.4.5, 6.4.6, 6.4.7, 6.4.8, 6.4.9, 6.4.10, 6.4.11, 6.5.0, 6.5.1, 6.5.2, 6.5.3, 6.5.4, 6.5.5, 6.5.6, 6.5.7, 6.5.8, 6.5.9, 6.5.10, 6.6.0, 6.6.1, 6.6.2, 6.6.3, 6.6.4, 6.6.5, 6.6.6, 6.6.7, 6.6.8, 6.6.9, 6.6.10, 6.6.11, 6.6.12