Distribute indexing and searching
This topic discusses the concepts and hardware requirements for distributing the indexing and searching components of your Splunk Enterprise deployment.
Concepts of distributed indexing and searching
Scale your Splunk Enterprise deployment by distributing searching and indexing across multiple machines. Indexers bring in, store, and search the data. Search heads manage search requests and present results.
Because indexers require much more disk I/O throughput than search heads do, give your environment more indexing capacity by reducing the overhead required for searching. The key points are as follows:
- When you add indexers to the deployment, the aggregate rate of data consumption and total available storage increases, which provides additional capacity to manage increased search loads.
- When you add search heads, you increase the number of concurrent users that the deployment can handle, thus providing capacity for increased search load.
The more search heads you add to the deployment, the faster you can find the data that you indexed.
Considerations for search performance versus indexing performance
While the two points in the previous section are best practice for improving indexing speed, some caveats apply, particularly when it comes to search speed.
As your indexers consume data, they store it in buckets, which are the individual elements of an index. As more data comes in, the number of buckets increases. A higher number of buckets can impact search speed because of the throughput required to navigate through those buckets for the data you want. This impact is noticeable for index buckets that hold smaller amounts of data.
As the number of buckets increases, the indexer must manage the buckets, which it does by "rolling" them to make room for new incoming data. This procedure takes up I/O cycles as well, which reduces the number of I/O cycles available to fetch events for search requests.
The key points to remember are:
- Adding search heads to your distributed deployment does not guarantee improved search performance. A mix of search heads and indexers is vital for performance increases.
- The number and types of search also impact indexer performance. Some search types tax an indexer's CPU, while others apply pressure to the disk subsystem.
See "Accommodate concurrent users and searches" in this manual for details on simultaneous searches.
Estimate your storage requirements
Accommodate many simultaneous searches
This documentation applies to the following versions of Splunk® Enterprise: 6.2.0, 6.2.1, 6.2.2, 6.2.3, 6.2.4, 6.2.5, 6.2.6, 6.2.7, 6.2.8, 6.2.9, 6.2.10, 6.2.11, 6.2.12, 6.2.13, 6.2.14, 6.2.15