How concurrent users and searches impact performance

The largest performance factor in a Splunk Enterprise deployment are:

• The number of concurrent users.
• The number of concurrent searches.
• The types of searches used.

A user that submits a search request will use one CPU core on each indexer until the search is complete. Any additional searches that the user submits also account for one CPU core. You can adjust the number of global concurrent searches that a machine can run. See Expected performance and known limitations of real-time searches and reports in the Splunk Enterprise Search manual.

The type of search a user invokes also impacts hardware resource usage. See How search types affect Splunk Enterprise performance.

How to maximize search performance

To accommodate the resource overhead of running many concurrent searches, add additional indexers, and maximize the physical memory available to the indexers. The indexers do the bulk of the work in search operations, such as identifying the data requested, reading the data from disk, decompressing it, filtering the data, and streaming the results.

For example, if a search uses 200MB of memory, and there are 48 concurrent search requests, that equates to about 10GB of memory to meet the search load not including the OS requirements. The amount of available memory is an important resource to monitor. While performance on an indexer declines gradually with increased CPU usage from concurrent search jobs, it drops dramatically when all available physical memory is exhausted.

Search performance: A basic scenario

The aggregate run time of all searches increases as the number of available CPU cores on the indexer decreases. For example, on an indexer with no load and 12 available cores, the first searches to arrive for processing can complete within a short period of time. For this scenario, all searches will run to completion within 10 seconds.

12 concurrent searches: One indexer with 12 cores and no data being indexed.

No. of concurrent searches	/ No. of avail. cores	= No. of searches per core	No. of sec. per individual search	= Approx. time (sec.) to complete all searches
12	12	1	10	10

When there are 48 searches to run concurrently, the total time to complete all searches increases significantly.

48 concurrent searches: One indexer with 12 cores and no data being indexed.

No. of concurrent searches	/ No. of avail. cores	= No. of searches per core	No. of sec. per individual search	= Approx. time (sec.) to complete all searches
48	12	4	10	40

Because the indexers do the bulk of the work in search operations, such as identifying the data requested, reading the data from disk, decompressing it, filtering some data, and streaming the results, it is best practice to add indexers to decrease the total amount of time to return all search results.

A deployment with more indexers provides an aggregate increase in cores to improve the search completion time when there are many concurrent searches. When there are more cores available than concurrent searches, the cores can be used by Splunk Enterprise to perform maintenance operations, or can remain idle.

12 concurrent searches: 4 indexers with 12 cores per indexer and no data being indexed.

No. of concurrent searches	/ No. of avail. cores	= No. of searches per core	No. of sec. per individual search	= Approx. time (sec.) to complete all searches
12	48	1. By default, a search cannot take advantage of multiple cores.	10	10

48 concurrent searches: 4 indexers with 12 cores per indexer and no data being indexed.

No. of concurrent searches	/ No. of avail. cores	= No. of searches per core	No. of sec. per individual search	= Approx. time (sec.) to complete all searches
48	48	1	10	10

Search performance: Indexing data scenario

In an active deployment, the system is not sitting idle while searches arrive. If an indexer ingests 150GB/day of data, then it will use up to 4 of the available cores for indexing processes. With fewer cores available, the time it takes to return all search results increases.

12 concurrent searches: One 12-core indexer, with 8 available cores.

No. of concurrent searches	/ No. of avail. cores	= No. of searches per core	No. of sec. per individual search	= Approx. time (sec.) to complete all searches
12	8	2, as each core will remain in use until the prior search completes.	10	20

48 concurrent searches: One 12-core indexer, with 8 available cores.

No. of concurrent searches	/ No. of avail. cores	= No. of searches per core	No. of sec. per individual search	= Approx. time (sec.) to complete all searches
48	8	6	10	60

48 concurrent searches: 4 12-core indexers with 8 available cores.

No. of concurrent searches	/ No. of avail. cores	= No. of searches per core	No. of sec. per individual search	= Approx. time (sec.) to complete all searches
48	32	2, as each core will remain in use until the prior search completes.	10	20

Having fewer indexers with greater core counts per indexer can decrease the amount of total time for search, but also reduces scaling efficiency by providing less aggregate IOPS for search operations.

Adding indexers reduces the indexing load on any one machine. Additionally, you reduce the search time, lower the impact of high search concurrency, and lower the impact of resource contention for I/O and memory.