Configure admission rules to prefilter searches

Admission rules let you filter out searches automatically before they start, based on a predicate (condition) that you define. If a search meets the specified condition, the search does not run.

You can use admission rules to prevent the execution of rogue searches, such as poorly written and potentially harmful searches that might consume an excessive amount of resources and interfere with critical search workloads. For example, you can create a rule to filter out wildcard searches that target all indexes, or filter out searches in the alltime time range.

You can also use admission rules to set up time-bound access to searches for roles, users, apps and so on. For example, you can create a rule that filters out all ad hoc searches from a certain role during peak business days, but allows the same role to run searches on weekends. And you can create rules that limit the number of concurrent ad hoc searches to retain search capacity for scheduled searches.

Admission rules have no explicit ordering. All admission rules are evaluated when a search is dispatched. If a search meets the conditions of a rule, the rule takes effect before the search is executed. If a search is already running, and you create a new admission rule that applies to that search, the running search is not affected by the new rule.

Admission rules apply to base searches only. Admission rules do not apply to subsearches.

Admission rules evaluation behavior

Admission rules have no explicit ordering. Admission rules are evaluated for ad hoc searches at the the time the search is dispatched.

Admission rules for scheduled searches are evaluated asynchronously every 600s by default. As a result, changes to an admission rule or scheduled search can take up to 600s to become active.

If a search meets the conditions of a rule, the rule takes effect before the search is executed. If a search is already running, and you create a new admission rule that applies to that search, the running search is not affected by the new rule.

Create an admission rule

You can create and edit admission rules using Splunk Web, the CLI or REST API.

When you create an admission rule, the configuration is stored under the [search_filter_rule:<rule_name>] stanza in workload_rules.conf. The default app context is search. Unless you specify a different app context, admission rules are stored along with workload rules in etc/apps/search/local/workload_rules.conf. For more information, see View workload_rules.conf.

Create an admission rule using Splunk Web

To create and edit admission rules, a user's role must have the list_workload_rules and edit_workload_rules capabilities.

To create an admission rule using Splunk Web:

1. In Splunk Web, click Settings > Workload Management > Admission Rules.
2. Click Add Admission Rule.
3. Define the following fields to configure a new admission rule:

   Field	Action
   Name	Specify the name of the admission rule.
   Predicate (Condition)	Specify a predicate (condition) that must match to trigger this rule. The predicate syntax is <type>=<value> with optional AND, OR, NOT, (). For example, app=search AND role=power triggers all searches belonging to both the Search app and the power role. Valid predicate types are app, role, index, user, search_type, search_mode, and search_time_range, and adhoc_search_percentage For supported predicate values, see Specify predicate values. In complex predicates, AND, OR, and NOT operators must be upper case. Lower case is not supported.
   Schedule	(Optional) Set a schedule for the admission rule. The schedule determines the time period during which the rule is valid. If set to Always On (the default), the rule remains valid indefinitely and does not expire. If set to Time Range, the rule is valid during the specified time range only and expires when the time range ends. If set to Every Day, Every Week, or Every Month, the rule becomes valid on a recurring basis during the specified time range every day, on the specified days of the week, or on the specified days of the month. The schedule time for an admission rule is based on the system timezone, regardless of the timezone set for an individual user in the UI.
   Action	Specify the action to perform when a search meets the predicate condition. If set to Filter search, the rule filters out any search that meets the specified predicate condition. If set to Queue search, the rule places a search that meets the predicate condition in a queue to be run at a later time, when the predicate condition allows it. This action is currently allowed for adhoc_search_percentage predicate type rules only.
   User Message	Enter a custom message that notifies the end user when a search triggers the admission rule action. For example, "This search meets specified admission rule conditions. The search was not executed." If an ad hoc search triggers the rule action, the custom message appears beneath the search bar in the Search and Reporting app. If a scheduled search triggers the action, a default message appears in scheduler.log only.

4. Click Submit.

Specify predicate values

The following table shows valid values for each admission rule predicate type:

Predicate type	Valid values
app	Name of the app. For example, app=search The correct name to specify for an app is the name of the app directory located in $SPLUNK_HOME/etc/apps. You can also find the correct name for an app in Splunk Web: Click Apps > Manage Apps. See app names listed under Folder name. App names are case insensitive.
role	Name of the role. For example, role=admin. The role predicate supports a wildcard (*) in the rule definition. For example, a rule that contains role=support_* will match the role names support_team1, support_team2 and so on. A rule that contains role=* will match all role name values. For information on how role inheritance impacts role predicate matches, see Searches run by single user can match multiple roles in the Workload Management manual in the Splunk Enterprise documentation.
user	Name of any valid user. For example, user=bob. The reserved internal user "nobody" is invalid; the reserved internal user "splunk-system-user" is valid. The user predicate supports a wildcard (*) in the rule definition. For example, a rule that contains user=* will match all user name values.
index	Name of the index. For example, index=_internal. Value can refer to internal or public index. The index predicate supports a wildcard (*) in the rule definition. For example, a rule that contains index=support_*, will match the index names support_prod, support_test, and so on. A rule that contains index=_* will match any index name that starts with an underscore, such as _internal, _audit, and _introspection. index=* is a special case, where * is not treated as a wildcard, but as a literal string match. For example, if a rule contains index=*, it will match the exact string index=* or index=_*. index=* will not match all internal indexes, such as index=_internal, index=_audit, and so on.
search_type	Supports adhoc, scheduled, datamodel_acceleration, report_acceleration, and summary_index search types.
search_mode	Supports realtime and historical search modes.
search_time_range	Supports alltime search time range only.
adhoc_search_percentage	[0-100]. The percentage of the total search concurrency limit that you want to allocate to ad hoc searches. Specify an adhoc_search_percentage value to set a limit on the number of ad hoc searches that can run concurrently on the local search head. Use this to prevent skipped scheduled searches on systems running large numbers of ad hoc searches. See Example 4: Ad hoc search quota control. When specifying adhoc_search_percentage, you must also specify search_type=adhoc in the predicate statement. For example, search_type=adhoc AND adhoc_search_percentage>10. In a search head cluster environment, the adhoc_search_percentage value applies to each individual cluster member. The adhoc_search_percentage predicate does not work with other predicate types, with the exception of search_type=adhoc. Specifying any another predicate type returns an error.

Create an admission rule using the CLI

To create an admission rule using the CLI, run the splunk add workload-rule command as follows, where predicate has the syntax <type>=<value> with optional AND, NOT, OR, (): and the value of action is filter:

./splunk add workload-rule <rule_name> -predicate <predicate_string> -action filter

To remove an admission rule, run the splunk remove workload-rule command where the value of workload_rule_type is search_filter:

./splunk remove workload-rule rule <rule_name> -workload_rule_type search_filter

To list admission rules:

./splunk list workload-rule -workload_rule_type search_filter

Create an admission rule using REST

Send a POST request to the following endpoint:

workloads/rules 

For example:

curl  -k -u <user:passwd> -X POST <host:mgmt_port>/services/workloads/rules -d name=<rule_name> -d predicate=<predicate_string> -d action=filter

For endpoint details, see workloads/rules in the REST API Reference Manual.

Enable admission rules

You must globally enable the admission rules feature for any existing rules to apply when a search is executed. You can enable or disable admission rules using Splunk Web, the CLI or REST API.

When you enable the admission rules feature, the setting is stored under the [search_admission_control] stanza in workload_policy.conf. This file is located by default in etc/apps/search/local. For more information, see workload_policy.conf.

To enable admission rules, a user's role must hold the list_workload_policy and edit_workload_policy capabilities.

Enable admission rules using Splunk Web

To enable or disable admission rules using Splunk Web:

1. In Splunk Web, click Settings > Workload Management > Admission Rules.
2. Toggle the switch to enable or disable Admission rules.

Enable admission rules using the CLI

To enable or disable admission rules, run the following CLI command:

./splunk edit search-admission-control -admission_rules_enabled <1|0>

Enable admission rules using REST

Send a POST request to the following endpoint:

workloads/policy/search_admission_control

For example:

curl -k -u <user:passwd> -X POST https://hostname:8089/services/workloads/policy/search_admission_control -d admission_rules_enabled=<1|0>

For endpoint details, see workloads/policy/search_admission_control in the REST API Reference Manual.

Enable individual admission rules

You can enable or disable individual admission rules. This lets you create and save multiple different admission rules and apply them as needed. Individual admission rules are enabled by default when you create them. Disabled admission rules are not evaluated and have no effect on search execution.

When you enable or disable an admission rule, the is setting is stored under the [search_filter_rule:<rule_name>] stanza for the individual rule in workload_rules.conf. For more information, see workload_rules.conf.

You can enable or disable individual admission rules using Splunk Web, the CLI, or REST API.

Enable individual admission rules using Splunk Web

To enable or disable an individual admission rule using Splunk Web:

1. In Splunk Web, click Settings > Workload Management > Admission Rules.
2. In the Status column, toggle the switch to enable or disable the individual admission rule.

Enable individual admission rules using CLI

To enable or disable an individual admission rule, run the following CLI command, where the value of -workload_rule_type is search_filter:

./splunk <enable|disable> workload-rule <rule_name> -workload_rule_type search_filter

Enable individual admission rules using REST

Send a POST request to the workloads/rules endpoint, as shown:

curl -k -u <user:pass> -X POST https://localhost:8089/services/workloads/rules/<rule_name>/<enable|disable> -d workload_rule_type=search_filter

For endpoint details, see workloads/rules in the REST API Reference Manual.

Example admission rules

The following examples show how you can define admission rules to filter out searches based on the objectives of your use case.

Example 1: Stop wildcard searches

The following rule excludes data model acceleration searches that use index=* from the filter:

index=* AND (NOT search_type=datamodel_acceleration)

Example 2: Stop all-time searches

Some monitoring searches use the alltime time range. The following rule excludes those searches from the filter:

search_time_range=alltime AND (NOT role=sc_admin) AND (NOT app=splunk_instance_monitoring)

The "View index inheritance for roles" feature also uses an alltime search. You can exclude users or roles that need to view indexes from the filter. The above rule excludes the sc_admin role.

REST API searches default to the alltime time range, unless you explicitly define a different time range in the search query.

Example 3: Do not allow certain users to run ad hoc searches at peak hours

search_type=adhoc AND role=new_users

After you define the admission rule predicate, set the schedule for the rule, specifying the time range that corresponds to your peak hours.

Example 4: Ad hoc search quota control

You can define admission rules to limit the number of concurrent ad hoc searches running on your system.

The following rule limits the number of concurrent ad hoc searches to 50% of the total search concurrency limit:

search_type=adhoc AND adhoc_search_percentage>50 

After you specify the predicate statement, set the action for the rule to Filter search or Queue search.

If set to Filter search, the rule filters out any ad hoc searches over the 50% concurrency limit. If set to Queue search, the rule places any ad hoc searches over the 50% concurrency limit in a queue to be run again when the number of concurrent ad hoc searches falls below the 50% threshold.

For more information on search concurrency limits in Splunk Enterprise, see Configure the priority of scheduled reports in the Reporting Manual.

For more admission rule use case examples, see Scenario 3: Create admission rules to prefilter searches.