Splunk® Enterprise

Distributed Deployment Manual

Download manual as PDF

This documentation does not apply to the most recent version of Splunk. Click here for the latest version.
Download topic as PDF

Accommodate concurrent users and searches

This topic discusses how to accommodate many simultaneous searches, search types, and concurrent users with your distributed Splunk Enterprise deployment.

Overview

The biggest performance impacts on a Splunk Enterprise deployment are the number of concurrent users, the number of concurrent searches, and the types of searches you employ during the course of the deployment's operation. Each of these elements affects the deployment in different ways.

How concurrent users and searches affect performance

When a user submits a search request, the search request may take up to one CPU core on each indexer to process while it is running. Any additional searches launched by that user also account for one CPU core.

The type of search the user invokes also impacts resource usage. The additional amount of CPU or disk usage varies depending on the search type. For additional information about search types, read "How Splunk Enterprise looks through your data" in this chapter.

How to maximize search performance

The best way to address the resource overhead for many concurrent searches is to configure the environment to handle search requests completely within available physical server memory. While this is important for the servers that you designate as search heads, it's particularly important for indexers in your deployment, as indexers must both index incoming data and search existing data.

For example, in a deployment that has up to 48 concurrent searches happening at a time, a single search that uses 200 MB of memory translates to nearly 10 GB of memory required to satisfy the 48 concurrent search requests. The amount of available memory is a very important statistic - while performance on an indexer only declines gradually with increased CPU usage from concurrent search jobs, it drops dramatically when the server exhausts all available physical memory. A single reference indexer could not handle 48 concurrent searches - at the very least, additional memory is required for satisfactory performance.

Notwithstanding the memory constraints, a search's run time increases proportionally as the number of free CPU cores on a system decreases. For example, on an idle system with 8 available cores, the first eight searches to arrive get serviced immediately by each core, and complete within a short period of time (for the purposes of this example, 10 seconds).

If there are 48 searches running concurrently, then completion time increases significantly, as shown in the calculation below:

No. of concurrent searches / No. of avail. cores = No. of searches per core x No. of sec. per individual search = Total time (sec.) per search
48 8 6 10 60

Since indexers do the bulk of the work in search operations (reading data off disk, decompressing it, extracting knowledge and reporting), it's best practice to add indexers to decrease the amount of time per search. In this example, to return to the performance level of 10-second searches, you must deploy 6 indexers with eight cores each:

No. of concurrent searches / No. of avail. cores = No. of cores per search x No. of sec. per individual search = Total time (sec.) per search
8 48 6 10 1.6
No. of concurrent searches / No. of avail. cores = No. of cores per search x No. of sec. per individual search = Total time (sec.) per search
48 48 1 10 10

Note: In this example, one search head to service search requests is okay. It might be appropriate, however, to set aside a second search head to create summary indexes.)

In many cases, though, the system isn't idle before searches arrive. For example, if an indexer is indexing 150 GB/day of data at peak times, then up to 4 of the 8 cores are in use indexing the data. In this case, search times increase significantly:

No. of concurrent searches / No. of avail. cores = No. of cores per search x No. of sec. per individual search = Total time (sec.) per search
4 4 1 10 10
No. of concurrent searches / No. of avail. cores = No. of searches per core x No. of sec. per individual search = Total time (sec.) per search
48 4 12 10 120

Increasing the number of cores per server does decrease the amount of time taken per search, but is not the most effective way to streamline search operations. One system with 16 cores has the following performance (again, assuming an idle system before searches arrive):

No. of concurrent searches / No. of avail. cores = No. of cores per search x No. of sec. per individual search = Total time (sec.) per search
16 16 1 10 10
No. of concurrent searches / No. of avail. cores = No. of searches per core x No. of sec. per individual search = Total time (sec.) per search
48 16 3 10 30

With two 8-core servers, the performance profile becomes the following:

No. of concurrent searches / No. of avail. cores = No. of cores per search x No. of sec. per individual search = Total time (sec.) per search
8 16 2 10 5
No. of concurrent searches / No. of avail. cores = No. of cores per search x No. of sec. per individual search = Total time (sec.) per search
16 16 1 10 10
No. of concurrent searches / No. of avail. cores = No. of searches per core x No. of sec. per individual search = Total time (sec.) per search
48 16 3 10 30

Two 8-core servers cost only slightly more than one 16-core server. In addition, two 8-core servers provide significantly more available disk throughput than one 16-core server does, based on spindle count. This is especially important for indexers because of the high disk bandwidth that they require.

Adding indexers reduces the indexing load on any system and frees CPU cores for searching. Also, since the performance of almost all types of search scales with the number of indexers, searches will be faster, which mitigates the effect of reduced performance from sharing resources amongst both indexing and searching. Increasing search speed reduces the chance of concurrent searches with concurrent users.

In real-world situations with hundreds of users, each user will run a search every few minutes, though not at the exact same time as other users. By adding indexers and reducing the search time, you reduce the concurrency factor and lower the concurrency-related I/O and memory contention.

PREVIOUS
Reference hardware
  NEXT
How Splunk apps affect resource requirements

This documentation applies to the following versions of Splunk® Enterprise: 6.0, 6.0.1, 6.0.2, 6.0.3, 6.0.4, 6.0.5, 6.0.6, 6.0.7, 6.0.8, 6.0.9, 6.0.10, 6.0.11, 6.0.12, 6.0.13, 6.0.14, 6.0.15, 6.1, 6.1.1, 6.1.2, 6.1.3, 6.1.4, 6.1.5, 6.1.6, 6.1.7, 6.1.8, 6.1.9, 6.1.10, 6.1.11, 6.1.12, 6.1.13, 6.1.14


Comments

Hi Supersleepwalker,<br /><br />It does make the chart kind of confusing, I agree. Unfortunately, doing so is the only way to:<br /><br />- Show the correlation between available cores and how concurrent searches impact the availability of those cores.<br />- Prevent the metric from being a decimal (and thus making the chart even harder to read).

Malmoore
January 30, 2014

Switching the third column back and forth between "= No. of cores per search" and "= No. of searches per core" (its reciprocal) makes this page hard to read. Your natural tendency is to look down the columns for an apples to apples comparison, and you can't with that column switching.

Supersleepwalker
January 30, 2014

Was this documentation topic helpful?

Enter your email address, and someone from the documentation team will respond to you:

Please provide your comments here. Ask a question or make a suggestion.

You must be logged into splunk.com in order to post comments. Log in now.

Please try to keep this discussion focused on the content covered in this documentation topic. If you have a more general question about Splunk functionality or are experiencing a difficulty with Splunk, consider posting a question to Splunkbase Answers.

0 out of 1000 Characters