Parallelization settings

New settings are available in Splunk Enterprise to improve search and indexing performance.

Who can use these settings

The parallelization settings are designed to improve the performance of specific components in Splunk Enterprise. The parallelization features are intended for customers with excess CPU cores and I/O capacity to leverage their hardware for improved performance across the indexing tier. You can use these settings to allocate CPU resources to the most common uses for your Splunk platform environment, tuning the indexers to meet that demand.

Summary of settings

Setting	Description
Batch mode search parallelization	Allows a batch mode search to open additional search pipelines on each indexer, processing multiple buckets simultaneously.
Parallel summarization for data models	Allows the scheduler to run concurrent data model acceleration searches on the indexers.
Parallel summarization for report accelerations	Allows the scheduler to run concurrent report acceleration searches on the indexers.
Index parallelization	Allows concurrent data processing pipelines on indexers and forwarders.

If the indexers in your Splunk platform environment exceed the reference hardware specifications, you may review the use case and increase one parallelization settings up to the maximum recommended value. If your indexers are at or near capacity, changing the parallelization settings can have a negative impact on search and indexing performance. All parallelization settings require a service restart to take effect.

Batch mode search parallelization

Batch mode searches are designed to search and return event data by bucket, instead of by time. By adding more batch search pipelines, multiple buckets are processed simultaneously, speeding the return of search results. Customers leveraging batch mode search parallelization can see a doubling of speed in returning batch mode search results.

Setting name	Default	Maximum recommended value	Impact
batch_search_max_pipeline	1	2	Multiplies the number of search pipelines per batch mode search, per indexer.

Adjusting the batch_search_max_pipeline setting in limits.conf to 2 multiplies the IO, processing, and memory used by batch mode searches on every indexer. A value of 2 provides the best performance increase, with higher values succumbing to diminishing returns. For configuration details, see Configure batch mode search parallelization in the Splunk Enterprise Knowledge Manager Manual.

Splunk administrators can use the monitoring console to monitor and track indexer resource use. For more details, see About the monitoring console in Monitoring Splunk Enterprise.

Parallel summarization

There are two types of accelerated searches: data model accelerations and report accelerations. Both acceleration types create search results on disk beside each index bucket. When a scheduled acceleration search is unable to keep up with the data volume in an index, latency is introduced into the search results. By allowing the scheduler to run concurrent acceleration searches on the indexers, multiple buckets are processed simultaneously, speeding the creation of accelerated search results. Customers leveraging parallel summarization can see a doubling of speed in building accelerated search results.

Data model accelerations

Setting name	Default	Maximum recommended value	Impact
acceleration.max_concurrent	3	3	Multiplies the number of scheduled acceleration searches per data model, per indexer.

The acceleration.max_concurrent setting in datamodels.conf defaults to 3, multiplying the IO, processing, and memory used while running scheduled acceleration searches on every indexer. A value of 3 provides the best performance increase, with higher values succumbing to diminishing returns. For configuration details, see Parallel Summarization in the Splunk Enterprise Knowledge Manager Manual

Report accelerations

Setting name	Default	Maximum recommended value	Impact
auto_summarize.max_concurrent	1	2	Multiplies the number of scheduled acceleration searches per search, per indexer.

Adjusting the auto_summarize.max_concurrent setting in savedsearches.conf to 2 multiplies the IO, processing, and memory used while running scheduled acceleration searches on every indexer. A value of 2 provides the best performance increase, with higher values succumbing to diminishing returns. For configuration details, see Use parallel summarization to speed up creation and maintenance of report summaries in the Splunk Enterprise Knowledge Manager Manual.

Splunk administrators can use the monitoring console to monitor and track indexer resource use. For more details, see About the monitoring console in Monitoring Splunk Enterprise.

Index parallelization

Index parallelization allows an indexer to maintain multiple pipeline sets. A pipeline set handles the processing of data, from receiving streams of events, through event processing, and writing the events to disk. By allowing an indexer to create and operate multiple pipelines, multiple data streams can be processed with additional CPU cores, accelerating data parsing and disk writing up to the limits of the indexer's I/O capacity. Customers leveraging index parallelization can see an increase in an indexer's sustained indexing load, or a doubling of indexing speed when receiving a sudden surge of data from the forwarders.

Setting name	Default	Maximum recommended value	Impact
parallelIngestionPipelines	1	2	Multiplies the number of pipelines per indexer.

Adjusting the parallelIngestionPipelines setting in server.conf to 2 will use an additional 4-6 CPU cores, and requires 300-400 IOPS to maintain indexing thruput on every indexer. Also, there are fewer CPU cores available for search processing. A value of 2 provides the best performance increase, with higher values succumbing to diminishing returns. For configuration details, see Manage pipeline sets for index parallelization in the Splunk Enterprise Managing Indexers and Clusters of Indexers Manual

Splunk administrators can use the monitoring console to monitor and track indexer resource use. For more details, see About the monitoring console in Monitoring Splunk Enterprise.