Control search concurrency on search head clusters

Controlling search concurrency on search head clusters requires an understanding of two aspects of job allocation:

• How the captain allocates saved search jobs to cluster members
• How the cluster handles concurrent search limits and quotas

How the captain allocates saved searches

The captain schedules saved search jobs, allocating them to the various cluster members according to load-based heuristics. Essentially, it attempts to assign each job to the member currently with the least search load.

The captain can allocate saved search jobs to itself. It does not, however, allocate scheduled real time searches to itself.

If a job fails on one member, the captain reassigns it to a different member. The captain reassigns the job only once, as multiple failures are unlikely to be resolvable without intervention on the part of the user. For example, a job with a bad search string will fail no matter how many times the cluster attempts to run it.

You can designate a member as "ad hoc only." In that case, the captain will not schedule jobs on it. You can also designate the captain functionality as "ad hoc only." The current captain then will never schedule jobs on itself. Since the role of captain can move among members, this setting ensures that captain functionality does not compete with scheduled searches. See Configure a cluster member to run ad hoc searches only.

For details of your cluster's scheduler delegation process, view the Search Head Clustering: Scheduler Delegation dashboard in the monitoring console. See Use the monitoring console to view search head cluster status.

See also:

• The scheduler concurrency limit
• How the captain determines whether to allow a scheduled search to run

How the cluster handles concurrent search limits and quotas

Each search head cluster member, like non-clustered search heads, adheres to these concurrent search limits:

• Maximum historical search limit. This limit determines the maximum number of historical searches (combined scheduled and ad hoc) that a cluster member can run concurrently. See The maximum historical search limit.
• Scheduler concurrency limit. This limit is the maximum number of scheduled search jobs that the scheduler can run concurrently, determined separately for each member of the cluster. This limit is a subset of the maximum historical search limit. In search head clustering, the captain serves as the job scheduler for all cluster members. See The scheduler concurrency limit.

To enforce the concurrent historical search limits, the cluster bases its behavior according to settings for these search quotas:

• User/role search quotas. These quotas enforce the maximum number of concurrent historical searches (combined scheduled and ad hoc) allowable for a specific user/role. These quotas are configured with srchJobsQuota and related settings in authorize.conf. See the authorize.conf spec file for details on all the settings that control these quotas.
• Overall search quota. This quota enforces the maximum number of historical searches that the cluster can run concurrently.
• Ad hoc search quota. This quota is optional and is configured as an admission rule. If set, it enforces the maximum number of ad hoc searches that the cluster can run concurrently.

The search head cluster enforces the user/role quotas, overall quota, and ad hoc quota on either a cluster-wide or a member-by-member basis. See How the cluster enforces historical search quotas and Ad hoc search control.

The maximum historical search limit

The max_hist_searches value specifies the maximum number of historical searches that can run concurrently on a search head. Historical searches include all searches that run against historical data; that is, both non-realtime scheduled searches and ad hoc searches. The max_hist_searches value is not a setting but rather a calculated value.

This value is used in determining historical search quotas. The search head cluster captain also uses max_hist_searches when it calculates how many scheduled searches can run concurrently. See The scheduler concurrency limit.

The total_search_concurrency_limit setting in limits.conf determines how max_hist_searches is calculated:

total_search_concurrency_limit = auto | <integer>

If you change total_search_concurrency_limit on one member, the change will replicate automatically to the other members in the cluster.

If the total_search_concurrency_limit is set to auto

If the total_search_concurrency_limit setting is set to "auto", max_hist_searches is derived from a formula based on the number of CPUs on each search head:

 max_hist_searches =  max_searches_per_cpu x number_of_cpus + base_max_searches

For example, if the search head has 96 CPUs and the value of max_searches_per_cpu is 2 and the value of base_max_searches setting is 6, the value of max_hist_searches is 198. In other words, the search head can run 198 concurrent historical searches:

 max_hist_searches = 2 x 96 + 6 = 198

The max_searches_per_cpu and base_max_searches settings are set in limits.conf. These settings must be identical across all members of the cluster, even though the number of CPUs on individual members can vary.

If the total_search_concurrency_limit is set to an integer value

To define the search concurrency limit independently of the total CPUs across the cluster, you can set total_search_concurrency_limit to an integer value.

If the total_search_concurrency_limit setting is set to an integer value, the captain calculates the max_hist_searches for each member, using this formula:

 max_hist_searches =  total_search_concurrency_limit * (number_of_cpus_on_member / number_of_cpus_on_all_available_members)

For example if total_search_concurrency_limit = 2000 in a cluster with 10 members in the "Up" state, each with 96 CPUs, the value of max_hist_searches for each member is 200:

2000 * (96 / (96 * 10))

If the number of CPU cores vary among members, then max_hist_searches will also vary. One member might have 100 maximum searches, while a better provisioned member has 200 maximum searches, and so on..

The captain reports each member's max_hist_searches limit back to the member.

No restart is needed after changing the value of total_search_concurrency_limit.

For information on determining the state of a member, see Show cluster status.

The scheduler concurrency limit

The scheduler for the entire cluster runs on the captain. To determine the scheduler concurrency limit for each available member, the captain takes the member's max_hist_searches value, derived from the number of CPU cores on the member. It then multiplies the result by max_searches_perc in limits.conf (default 50%) to determine the scheduler limit:

max_hist_searches * max_searches_perc

Note:

• The max_hist_searches value specifies the maximum number of historical searches that can run concurrently on a search head. Historical searches include all searches that run against historical data; that is, both non-realtime scheduled searches and ad hoc searches. For information on how max_hist_searches is calculated, see The maximum historical search limit.
• By employing a percentage against the maximum number of allowable historical searches, this method ensures that scheduled searches do not consume all available search capacity. At a minimum, the remaining percentage of search capacity is available for running ad hoc searches.

To determine the cluster-wide scheduler concurrency limit, the captain adds the scheduler limits for each member available to run scheduled searches. The number of available members can be less than the total number of members.

Members available to run scheduled searches are members that are both:

• In the "Up" state. For information on determining the state of a member, see Show cluster status.
• Not configured as "ad hoc only".For information on "ad hoc only" members, see Configure a cluster member to run ad hoc searches only.

For example, assume a seven-member cluster in which all seven members are "Up" but two members are configured as "ad hoc only," This configuration results in five available members.

How the cluster enforces historical search quotas

Although each historical search quota type (user/role or overall) has its own attribute for setting its enforcement behavior, the behavior itself works the same for each quota type.

If you configure the cluster to enforce quotas on a member-by-member basis, each individual member uses its concurrent historical search quota (max_hist_searches) to determine whether to allow a search to run. No cluster-wide enforcement of searches occurs.

If you configure the cluster to enforce quotas on a cluster-wide basis, the captain determines the search quota by adding together the concurrent historical search quotas (max_hist_searches) for all members in the "Up" state. This number includes all "Up" members that are capable of running searches, including those configured as "ad hoc only."

The captain uses the computed cluster-wide quota to determine whether to allow a scheduled search to run. No member-specific enforcement of searches occurs, except potentially in the case of ad hoc searches, as described in Ad hoc search control.

In the case of user/role quotas, the captain multiplies the base concurrent search quota allocated to a user/role by the number of "Up" cluster members to determine the cluster-wide quota for that user/role. For example, in a seven-member cluster, it multiplies the value of srchJobsQuota by 7 to determine the number of concurrent historical searches for the user/role.

Note that a search running on a member will also fail if srchJobsQuota or srchDiskQuota is exceeded for the user on that member.

In the case of overall search quotas, the captain adds together the overall search quotas for all "Up" members to determine the cluster-wide quota for all searches.

When determining the number of cluster-wide concurrent searches, the captain includes both scheduled searches and ad hoc searches running on all members. The captain stops a scheduled search from running if it will cause the number of concurrent searches to exceed the cluster-wide search quota. It will stop an ad hoc search from running if the captain is controlling ad hoc search behavior centrally. See Ad hoc search control.

For details of your cluster's search concurrency status, view the Search Head Clustering: Status and Configuration dashboard in the monitoring console. See Use the monitoring console to view search head cluster status.

How the captain determines whether to allow a scheduled search to run

When determining whether to allow a historical scheduled search to run, the scheduler on the captain follows this order:

1. Does the search exceed the scheduler concurrency limit?
   If so, the search does not run.
2. In the case of cluster-wide enforcement only, does the search exceed the cluster-wide user/role historical search quota for the user/role running the search?
   If so, the search does not run.
3. In the case of cluster-wide enforcement only, does the search exceed the overall historical search quota?
   If so, the search does not run.

Note: The captain always controls, at least to some extent (as described above), the running of scheduled searches. It might also control whether ad hoc searches run, as described in Ad hoc search control. However, whether or not the captain controls the running of ad hoc searches, the members always feed information on their current ad hoc searches to the captain, which includes those searches when comparing concurrent searches against the quotas. See Cluster-wide historical search quotas and ad hoc searches.

Configure historical search quota enforcement behavior

You configure user/role-based quota enforcement behavior separately from overall search quota enforcement behavior.

All cluster members must use the same set of values.

Configure user/role-based quota enforcement behavior

Configure user/role-based quota enforcement behavior with the shc_role_quota_enforcement setting, under the [scheduler] stanza in limits.conf.

To enforce these quotas on a member-by-member basis, leave this attribute set to false, its default value.

To enforce these quotas on a cluster-wide basis instead, set the attribute to true:

shc_role_quota_enforcement=true

For details of this setting, see limits.conf.

Configure overall search quota enforcement behavior

Configure overall search quota enforcement behavior with the shc_syswide_quota_enforcement setting, under the [scheduler] stanza in limits.conf.

To enforce this quota on a member-by-member basis, leave this attribute set to false, its default value.

To enforce this quota on a cluster-wide basis instead, set the attribute to true:

shc_syswide_quota_enforcement=true

For details of this setting, see limits.conf.

Ad hoc search control

Ad hoc search concurrency is limited, by default, to the number of available slots for historical searches, minus the number of slots currently occupied by scheduled searches. Since historical searches include both scheduled and ad hoc searches, ad hoc searches can, in effect, use all slots not currently in use by saved searches. However, as described in the note that immediately follows, this behavior can be overridden so as to allow you to independently limit the number of concurrent ad hoc searches.

Note: Although, by default, ad hoc searches can occupy any search slots not currently in use by scheduled searches, you can create an admission rule that limits the number of concurrent ad hoc searches based on a percentage of the historical search limit. See Limit the number of concurrent ad hoc searches.

If all slots for ad hoc searches are currently in use, the search head queues or rejects the new ad hoc search.

Note: Whether a search head decides to queue or reject the search depends primarily on the default_allow_queue setting in limits.conf. The default value of "true" causes the search to be queued. If you are employing an ad hoc search quota admision rule, the rule's action also determines whether the search is queued or rejected. See Limit the number of concurrent ad hoc searches.

When determining whether a slot is available for an ad hoc search, the search head looks at both overall search quota and user/role search quotas. For example, a search slot might be available based on the overall search quota but not available for the particular user originating the ad hoc search.

Ad hoc search enforcement scope

In a search head cluster, ad hoc search control can be enforced either cluster-wide or on a per-member basis.

When enforcement is per-member, the search head receiving, or "originating", the ad hoc search request looks only at its own capacity in light of search quotas to decide whether to run the search.

When enforcement is cluster-wide, the search head originating the ad hoc search request informs the captain of the request:

• If the originating member has capacity, the captain tells the originating member to run the search. The originating member then runs the search and returns the results to its user.

• If the originating member does not have capacity, the captain attempts to identify another member with available capacity. If it identifies a member with capacity, the captain provides the originating member with the identity of the available member. The originating member proxies the search to the available member, which runs the search and returns the results to the originating member. The originating member then serves the results to the user.

Another enforcement option, called "overflow", provides a compromise between the two other options for performance reasons. If the originating member has capacity, it runs the search directly without involving the captain. Otherwise, it sends the search to the captain for allocation to another member.

Set the ad hoc enforcement option

The ad hoc enforcement seting is located in limits.conf. It has this syntax:

shc_adhoc_quota_enforcement = on | off | overflow

The values have these effects:

• "on" causes the ad hoc search admission process to be managed cluster-wide by the captain.
• "off" causes the ad hoc search admission process to be managed locally, by each member when it receives an ad hoc search request.
• "overflow" causes the originating member to first check its local capacity. If the member has capacity (that is, if the member is below its local limits for concurrent search quotas), it runs the search locally. If the member has reached its limit on concurrent searches, it sends the search to the captain for allocation to a member with capacity.

Note the following:

• The default value is "off".
• You do not need to restart the member after changing the value of shc_adhoc_quota_enforcement.
• If you change shc_adhoc_quota_enforcement on one member, the change will replicate automatically to the other members in the cluster.

Cluster-wide historical search quotas and locally managed ad hoc searches

Each historical search quota includes both scheduled searches and ad hoc searches. Because of the way that the captain learns about locally managed ad hoc searches, the number of cluster-wide concurrent searches can sometimes exceed the search quota. This is true for both types of search quotas, user/role quotas and overall quotas.

Ad hoc searches are considered locally managed when shc_adhoc_quota_enforcement is set to "off". They can also be locally managed when shc_adhoc_quota_enforcement is set to "overflow" and the originating member handles the ad hoc search.

If, for example, you configure the cluster to enforce the overall search quota on a cluster-wide basis, the captain handles quota enforcement by comparing the total number of searches running across all members to the search quota.

So, to enforce quotas, the captain must know two values:

• The overall historical search quota
• The number of concurrent searches running across all members

The captain calculates the overall historical search quota by adding together the concurrent search quotas for all "Up" cluster members, as described in How the cluster enforces historical search quotas.

The captain calculates the number of concurrent searches running across all members by adding together the total number of scheduled and ad hoc searches in progress:

• For scheduled searches, the captain always knows the number of concurrent scheduled searches, because it controls the search scheduling operation.
• For locally managed ad hoc searches, the captain depends on reporting from the individual members. When a new ad hoc search starts, the member running the search informs the captain, and the captain adds that search to the total concurrent search number.

When the number of all searches, both scheduled and ad hoc, reaches the quota, the captain stops initiating new scheduled searches until the number of searches falls below the quota.

A user always initiates an ad hoc search directly on a member. In the case of locally managed ad hoc searches, the member uses its own set of search quotas, without consideration or knowledge of the cluster-wide search quota, to decide whether to allow the search. The member then reports the new ad hoc search to the captain. If the captain has already reached the cluster-wide quota, then a new ad hoc search causes the cluster to temporarily exceed the quota. This results in the captain reporting more searches than the number allowable by the search quota.

Limit the number of concurrent ad hoc searches

You can optionally limit the number of concurrent ad hoc searches by setting an ad hoc search quota admission rule through use of the adhoc_search_percentage predicate. The ad hoc search quota admission rule operates independently of the other search concurrency settings configured through limits.conf,

For example, to limit new ad hoc searches when the number of concurrent ad hoc searches exceeds 50% of all searches, set an admission rule with the predicate search_type=adhoc AND adhoc_search_percentage>50.

You can either filter or queue ad hoc searches exceeding the quota by setting the rule's action to "filter" or "queue". If a search is filtered, it is dropped/rejected. If a search is queued, it will be retried later, once the number of concurrent ad hoc searches dips below the quota enforced by the rule.

Note: In order for a "queue" action to apply, the default_allow_queue setting in limits.conf must be set to its default value of "true". See Ad hoc search queue.

For details on how to configure and use admission rules, including the ad hoc search quota rule, see Configure admission rules to prefilter searches in the Workload Management manual.

Calculation of ad hoc search concurrency against the adhoc_search_percentage value is performed on a node-by-node basis if shc_adhoc_quota_enforcement in limits.conf is set to "off" or "overflow". The calculation is performed on a cluster-wide basis if shc_adhoc_quota_enforcement is set to "on".

The ad hoc search quota rule, if set, is layered on top of the other search concurrency limits configured in limits.conf.Note that, if you limit the number of concurrent scheduled searches to 50% through max_searches_perc in limits.conf and then limit the number of concurrent ad hoc searches to, say, 40% through this admission rule, the total number of concurrent historical searches will never exceed approximately 90% of total search capacity as represented by the max_hist_searches value.