Configure workload rules

Workload rules provide an automated method for assigning searches to workload pools and monitoring running searches. The rules are evaluated in the order in which they are listed. If a search meets the predicate condition defined in a rule, a specified action is taken. Workload rules are evaluated for every new search and reevaluated every 10 seconds.

There are two types of workload rules:

• Search placement rules
• Search monitoring rules

Search placement rules determine the pool in which a search is placed when you start a search. Predicates that you can define to control search placement include app, role, user, index, search_type, search_mode, and search_time_range. You can use search placement rules to ensure that high-priority searches are assigned to pools that provide adequate resources, while low-priority searches are restricted.

Search monitoring rules automatically trigger actions on running searches based on the defined rule predicate and the status of the search. When you create a monitoring rule, you must specify a runtime value in the predicate. If a search exceeds the runtime value, workload management performs the specified action. Supported actions are Abort search, Display in Messages, and Move search to alternate Pool. You can use monitoring rules to manage heavy search loads and prevent rogue processes from monopolizing pool resources.

Create a workload rule

You can create and edit workload rules using Splunk Web, the CLI, or REST API. Workload rule configurations are stored in workload_rules.conf. See View workload_rules.conf.

Create a workload rule in Splunk Web

1. In Splunk Web, click Settings > Workload Management.
2. Click Add Workload Rule.
3. Define the following fields to configure a new workload rule:

   Field	Action
   Name	Specify the name of the workload rule.
   Predicate (Condition)	Specify a predicate (condition) that must match for a search to access the specified workload pool. The predicate syntax is <type>=<value> with optional AND, OR, NOT, (). For example, app=search AND role=power maps all searches belonging to the Search app executed by a user that belongs to the power role to the corresponding workload pool. Valid predicate types are app, role, index, user, search_type, search_mode, search_time_range, and runtime. For supported predicate values, see Predicate type values in the next section on this page. In complex predicates, AND, OR, and NOT operators must be upper case. Lower case is not supported.
   Schedule	(Optional) Set a schedule for the workload rule. The schedule determines the time period during which the rule is valid. If set to Always On (the default), the rule remains valid indefinitely and does not expire. If set to Time Range, the rule is valid during the specified time range only and expires when the time range ends. If set to Every Day, Every Week, or Every Month, the rule becomes valid on a recurring basis during the specified time range every day, on the specified days of the week, or on the specified days of the month. The schedule time for a workload rule is based on the system timezone, regardless of the timezone set for an individual user in the UI.
   Action	Specify the action to perform when a search meets the predicate condition. Place search in a Pool (the default) assigns searches that meet the predicate condition to the specified workload pool. Abort search kills the search process. Display in Messages shows a message in Splunk Web to users that have all of the following required capabilities: list_workload_pools, edit_workload_pools, list_workload_rules, and select_workload_rules. Move search to alternate Pool moves the running search to a different specified pool. Abort search, Display in Messages, and Move search to alternate Pool actions apply to in-progress searches only. You must specify a runtime condition to enable these actions. For example, the predicate index=_internal AND runtime>1m triggers the specified action on all searches that contain index=_internal and run for more than one minute. The Place search in a pool action is not valid with rules containing a runtime condition. Abort search, Display in Messages, and Move search to alternate Pool actions are valid only when a runtime condition exists.
   Workload Pool	Select the workload pool to which this rule applies.

4. Click Submit.

Predicate type values

The following table shows valid predicate type values for workload rules:

Predicate type	Valid values
app	Name of the app. For example, app=search The correct name to specify for an app is the name of the app directory located in $SPLUNK_HOME/etc/apps. You can also find the correct name for an app in Splunk Web: Click Apps > Manage Apps. See app names listed under Folder name. App names are case insensitive.
role	Name of the role. For example, role=admin. For important details on role predicate values, see Searches run by single user can match multiple roles directly after this table on this page.
index	Name of the index. For example, index=_internal. Value can refer to internal or public index. You can optionally specify index=* to classify searches containing either index=* or index=_*.
user	Name of any valid user. For example, user=bob. The reserved internal user "nobody" is invalid; the reserved internal user "splunk-system-user" is valid.
search_type	adhoc, scheduled, datamodel_acceleration, report_acceleration, and summary_index
search_mode	realtime and historical
search_time_range	Supports alltime time range only.
runtime	The amount of time that a search must run in a workload pool to trigger a specified action, such as Abort search, Display in Messages, or Move search to alternate Pool. For example, if you specify runtime>1m, any search in the pool that runs for more than 1 minute is subject to the specified action. By default, runtime applies to both historical and real-time searches. If you want the runtime condition to apply to historical searches only or real-time searches only, you must specify the search_mode in the predicate condition. For example, if you specify runtime>5m AND search_mode=realtime, only those realtime searches in the pool that run for more than 5 minutes are subject to the specified action. Valid units for runtime values include s, second, seconds, m, minute, minutes, and h, hour, hours.

Searches run by a single user can match multiple roles

By default, the admin role in Splunk Enterprise inherits the user role and the power role. As a result of this role inheritance, searches run by a single user can match to multiple roles in workload rules. For example, searches run by the "admin" user can match to workload rules that specify role=admin, role=user and role=power.

To avoid having a single user, such as the "admin" user, match to multiple roles in your workload rules, specify user=<username> in the rule predicate as an alternative to specifying role.

For more information on role inheritance, see About configuring role-based user access in Securing Splunk Enterprise.

For more information on workload rule configuration settings, see workload_rules.conf in the Admin Manual.

Create a workload rule using the CLI

To create a workload rule using the CLI, run the splunk add workload-rule command as follows, where predicate has the syntax <type>=<value> with optional AND, NOT, OR, ():

./splunk add workload-rule <rule_name> -predicate <predicate> -workload_pool <pool_name>

For example:

./splunk add workload-rule "my_role_rule" -predicate "app=search AND (NOT index=_internal)" \
-workload_pool "pool_a"

To create a workload monitoring rule, the -predicate value must include a runtime condition, and specify the -action value. For the "move" action, you must also specify the destination -workload_pool value. For example:

./splunk add workload-rule "my_monitoring_rule" \
-predicate "(role!=admin AND (search_time_range=alltime OR index=*)) AND runtime>10m" \
-action <abort|move> -workload_pool "pool_b"

It is a best practice to use parentheses to group predicate conditions for clarity.

Create a workload rule using REST

Send a POST request to the following endpoint:

workloads/rules 

For endpoint details, see workloads/rules in the REST API Reference Manual.

View workload_rules.conf

When you create a workload rule using Splunk Web, the CLI, or REST, the rule configuration is stored in $SPLUNK_HOME/etc/apps/<app_name>/local/workload_rules.conf.

The default app context is search. Unless you specify a different app context, the rule is stored in etc/apps/search/local/workload_rules.conf.

It is not advisable to manually edit workload_rules.conf. If a manual edit of workload_rules.conf does occur, you must reload the file. Restart is not required.

workload_rules.conf contains workload rule definitions that determine the order in which rules are evaluated, the mapping of rules to workload pools, the schedule for rules, and workload monitoring rules and actions. For example:

[workload_rules_order]
rules = my_analyst_rule,my_app_rule,my_user_rule,my_index_rule,my_monitoring_rule

[workload_rule:my_app_rule]
predicate = app=search
workload_pool = my_app_pool

[workload_rule:my_analyst_rule]
predicate = role=analyst
workload_pool = my_analyst_pool
schedule = always_on

[workload_rule:my_user_rule]
predicate = user=admin
workload_pool = my_user_pool
schedule = always_on

[workload_rule:my_index_rule]
predicate = index=_internal
workload_pool = my_index_pool
schedule = time_range
start_time = 2019-01-01T04:00:00-08:00
end_time = 2019-01-05T04:00:00-08:00

[workload_rule:my_search_type_rule]
predicate = search_type=adhoc
workload_pool = my_adhoc_pool
schedule = every_day
start_time = 10
end_time = 15

[workload_rule:my_logical_rule_1]
predicate = app=search AND (NOT index=_internal)
workload_pool = my_logical_pool_1
schedule = every_week
start_time = 10
end_time = 23
every_week_days = [0,4,6]

[workload_rule:my_logical_rule_2]
predicate = NOT role=power OR user=admin
workload_pool = my_logical_pool_2
schedule = every_month
start_time = 1
end_time = 2
every_month_days = [1,5,16,31]

[workload_rule:my_monitoring_rule]
action = move
predicate = app=search AND runtime>1m
workload_pool = my_overflow_pool

For more information on workload rule settings, see workload_rules.conf.