Parallelization settings
New settings are available in Splunk Enterprise to improve search and indexing performance.
Who can use these settings
The parallelization settings are designed to improve the performance of specific components in Splunk Enterprise. The parallelization features are intended for customers with excess CPU cores and I/O capacity to leverage their hardware for improved performance across the indexing tier. You can use these settings to allocate CPU resources to the most common uses for your Splunk platform environment, tuning the indexers to meet that demand.
Summary of settings
Setting | Description |
---|---|
Batch mode search parallelization | Allows a batch mode search to open additional search pipelines on each indexer, processing multiple buckets simultaneously. |
Parallel summarization for data models | Allows the scheduler to run concurrent data model acceleration searches on the indexers. |
Parallel summarization for report accelerations | Allows the scheduler to run concurrent report acceleration searches on the indexers. |
Index parallelization | Allows concurrent data processing pipelines on indexers and forwarders. |
If the indexers in your Splunk platform environment exceed the reference hardware specifications, you may review the use case and increase one parallelization settings up to the maximum recommended value. If your indexers are at or near capacity, changing the parallelization settings can have a negative impact on search and indexing performance. All parallelization settings require a service restart to take effect.
Batch mode search parallelization
Batch mode searches are designed to search and return event data by bucket, instead of by time. By adding more batch search pipelines, multiple buckets are processed simultaneously, speeding the return of search results. Customers leveraging batch mode search parallelization can see a doubling of speed in returning batch mode search results.
Setting name | Default | Maximum recommended value | Impact |
---|---|---|---|
batch_search_max_pipeline |
1 | 2 | Multiplies the number of search pipelines per batch mode search, per indexer. |
Adjusting the batch_search_max_pipeline
setting in limits.conf
to 2 multiplies the IO, processing, and memory used by batch mode searches on every indexer. A value of 2 provides the best performance increase, with higher values succumbing to diminishing returns. For configuration details, see Configure batch mode search parallelization in the Splunk Enterprise Knowledge Manager Manual.
Splunk administrators can use the monitoring console to monitor and track indexer resource use. For more details, see About the monitoring console in Monitoring Splunk Enterprise.
Parallel summarization
There are two types of accelerated searches: data model accelerations and report accelerations. Both acceleration types create search results on disk beside each index bucket. When a scheduled acceleration search is unable to keep up with the data volume in an index, latency is introduced into the search results. By allowing the scheduler to run concurrent acceleration searches on the indexers, multiple buckets are processed simultaneously, speeding the creation of accelerated search results. Customers leveraging parallel summarization can see a doubling of speed in building accelerated search results.
Setting name | Default | Maximum recommended value | Impact |
---|---|---|---|
acceleration.max_concurrent |
3 | 3 | Multiplies the number of scheduled acceleration searches per data model, per indexer. |
The acceleration.max_concurrent
setting in datamodels.conf
defaults to 3, multiplying the IO, processing, and memory used while running scheduled acceleration searches on every indexer. A value of 3 provides the best performance increase, with higher values succumbing to diminishing returns. For configuration details, see Parallel Summarization in the Splunk Enterprise Knowledge Manager Manual
Setting name | Default | Maximum recommended value | Impact |
---|---|---|---|
auto_summarize.max_concurrent |
1 | 2 | Multiplies the number of scheduled acceleration searches per search, per indexer. |
Adjusting the auto_summarize.max_concurrent
setting in savedsearches.conf
to 2 multiplies the IO, processing, and memory used while running scheduled acceleration searches on every indexer. A value of 2 provides the best performance increase, with higher values succumbing to diminishing returns. For configuration details, see Use parallel summarization to speed up creation and maintenance of report summaries in the Splunk Enterprise Knowledge Manager Manual.
Splunk administrators can use the monitoring console to monitor and track indexer resource use. For more details, see About the monitoring console in Monitoring Splunk Enterprise.
Index parallelization
Index parallelization allows an indexer to maintain multiple pipeline sets. A pipeline set handles the processing of data, from receiving streams of events, through event processing, and writing the events to disk. By allowing an indexer to create and operate multiple pipelines, multiple data streams can be processed with additional CPU cores, accelerating data parsing and disk writing up to the limits of the indexer's I/O capacity. Customers leveraging index parallelization can see an increase in an indexer's sustained indexing load, or a doubling of indexing speed when receiving a sudden surge of data from the forwarders.
Setting name | Default | Maximum recommended value | Impact |
---|---|---|---|
parallelIngestionPipelines |
1 | 2 | Multiplies the number of pipelines per indexer. |
Adjusting the parallelIngestionPipelines
setting in server.conf
to 2 will use an additional 4-6 CPU cores, and requires 300-400 IOPS to maintain indexing thruput on every indexer. Also, there are fewer CPU cores available for search processing. A value of 2 provides the best performance increase, with higher values succumbing to diminishing returns. For configuration details, see Manage pipeline sets for index parallelization in the Splunk Enterprise Managing Indexers and Clusters of Indexers Manual
Splunk administrators can use the monitoring console to monitor and track indexer resource use. For more details, see About the monitoring console in Monitoring Splunk Enterprise.
Forwarder-to-indexer ratios |
This documentation applies to the following versions of Splunk® Enterprise: 7.0.0, 7.0.1, 7.0.2, 7.0.3, 7.0.4, 7.0.5, 7.0.6, 7.0.7, 7.0.8, 7.0.9, 7.0.10, 7.0.11, 7.0.13, 7.1.0, 7.1.1, 7.1.2, 7.1.3, 7.1.4, 7.1.5, 7.1.6, 7.1.7, 7.1.8, 7.1.9, 7.1.10, 7.2.0, 7.2.1, 7.2.2, 7.2.3, 7.2.4, 7.2.5, 7.2.6, 7.2.7, 7.2.8, 7.2.9, 7.2.10, 7.3.0, 7.3.1, 7.3.2, 7.3.3, 7.3.4, 7.3.5, 7.3.6, 7.3.7, 7.3.8, 7.3.9, 8.0.0, 8.0.1, 8.0.2, 8.0.3, 8.0.4, 8.0.5, 8.0.6, 8.0.7, 8.0.8, 8.0.9, 8.0.10, 8.1.0, 8.1.1, 8.1.2, 8.1.3, 8.1.4, 8.1.5, 8.1.6, 8.1.7, 8.1.8, 8.1.9, 8.1.10, 8.1.11, 8.1.12, 8.1.13, 8.1.14, 8.2.0, 8.2.1, 8.2.2, 8.2.3, 8.2.4, 8.2.5, 8.2.6, 8.2.7, 8.2.8, 8.2.9, 8.2.10, 8.2.11, 8.2.12, 9.0.0, 9.0.1, 9.0.2, 9.0.3, 9.0.4, 9.0.5, 9.0.6, 9.0.7, 9.0.8, 9.0.9, 9.0.10, 9.1.0, 9.1.1, 9.1.2, 9.1.3, 9.1.4, 9.1.5, 9.1.6, 9.2.0, 9.2.1, 9.2.2, 9.2.3, 9.3.0, 9.3.1
Feedback submitted, thanks!