Distribute indexing and searching
This topic discusses the concepts and hardware requirements for distributing the indexing and searching components of your Splunk Enterprise deployment.
Concepts of distributed indexing and searching
You scale your Splunk Enterprise deployment by dedicating searching and indexing across multiple servers. Indexers bring in, store, and search the data. Search heads manage search requests and present results.
Since indexers require much more disk I/O throughput than search heads do, you give your environment more indexing capacity by reducing the overhead required for searching. The key points to remember are:
- The more indexers you add to the deployment, the faster data is consumed and prepared for searches.
- The more search heads you add to the deployment, the faster you are able to find the data you indexed.
Considerations for search performance vs. indexing performance
While the two points shown above are best practice for improving indexing speed, there are some important caveats to note as well, particularly when it comes to search speed.
As your indexers consume data, they store it in buckets - individual elements of an index. As more data comes in, the number of buckets increases. An increased number of buckets - particularly those which hold smaller amounts of data - can impact search speed because of the throughput required to navigate through those buckets for the data that you're searching.
Additionally, as the number of buckets increases, the indexer must manage the buckets. It does this by "rolling" buckets - thus making room for new incoming data. This procedure takes up I/O cycles as well - cycles that could be used to fetch events for search requests.
The key points to understand are:
- You can't necessarily improve search performance simply by adding search heads to your distributed deployment. A mix of search heads and indexers is vital.
- The number and types of search also impact indexer performance. Some search types tax an indexer's CPU, others apply pressure to the disk subsystem.
More detail about how to plan for simultaneous searches is found in "Accommodate concurrent users and searches" in this manual.]
Hardware capacity planning for a distributed Splunk Enterprise deployment
How Splunk Enterprise looks through your data
This documentation applies to the following versions of Splunk® Enterprise: 5.0, 5.0.1, 5.0.2, 5.0.3, 5.0.4, 5.0.5, 5.0.6, 5.0.7, 5.0.8, 5.0.9, 5.0.10, 5.0.11, 5.0.12, 5.0.13, 5.0.14, 5.0.15, 5.0.16, 5.0.17, 5.0.18, 6.0, 6.0.1, 6.0.2, 6.0.3, 6.0.4, 6.0.5, 6.0.6, 6.0.7, 6.0.8, 6.0.9, 6.0.10, 6.0.11, 6.0.12, 6.0.13, 6.0.14, 6.0.15, 6.1, 6.1.1, 6.1.2, 6.1.3, 6.1.4, 6.1.5, 6.1.6, 6.1.7, 6.1.8, 6.1.9, 6.1.10, 6.1.11, 6.1.12, 6.1.13, 6.1.14