Configure workload rules
Workload rules provide an automated way to assign searches to workload pools and monitor running searches. Workload management evaluates workload rules in the order in which they are listed. If a search meets the predicate condition defined in a rule, the specified action is taken. Workload rules are evaluated for every new search and reevaluated every 10 seconds.
There are two types of workload rules:
- Search placement rules
- Search monitoring rules
Search placement rules determine the pool in which a search is placed when you start a search. Predicates that you can define to control search placement include app
, role
, user
, index
, search_type
, search_mode
, and search_time_range
. You can use search placement rules to ensure that high-priority searches are assigned to pools that provide adequate resources, while low-priority searches are restricted.
Search monitoring rules automatically trigger actions on running searches based on the defined rule predicate and the status of the search. When you create a monitoring rule, you must specify a runtime
value in the predicate. If a search exceeds the runtime
value, workload management performs the specified action. Supported actions are Abort search
, Display in Messages
, and Move search to alternate Pool
. You can use monitoring rules to manage heavy search loads and prevent rogue processes from monopolizing pool resources.
Create a workload rule
You can create and edit workload rules using Splunk Web, the CLI, or REST API. Workload rule configurations are stored in workload_rules.conf
. See View workload_rules.conf.
Create a workload rule in Splunk Web
- In Splunk Web, click Settings > Workload Management.
- Click Add Workload Rule.
- Define the following fields to configure a new workload rule:
Field Action Name Specify the name of the workload rule. Predicate (Condition) Specify a predicate (condition) that must match for a search to access the specified workload pool. The predicate syntax is <type>=<value> with optional AND, OR, NOT, (). For example, app=search AND role=power
maps all searches belonging to the Search app executed by a user that belongs to the power role to the corresponding workload pool.
Valid predicate types are
app
,role
,index
,user
,search_type
,search_mode
,search_time_range
, andruntime
.
For supported predicate values, see Specify predicate values in the next section on this page.
In complex predicates,AND
,OR
, andNOT
operators must be upper case. Lower case is not supported.Schedule (Optional) Set a schedule for the workload rule. The schedule determines the time period during which the rule is valid.
If set toAlways On
(the default), the rule remains valid indefinitely and does not expire.
If set toTime Range
, the rule is valid during the specified time range only and expires when the time range ends.
If set toEvery Day
,Every Week
, orEvery Month
, the rule becomes valid on a recurring basis during the specified time range every day, on the specified days of the week, or on the specified days of the month.
The schedule time for a workload rule is based on the system timezone, regardless of the timezone set for an individual user in the UI.Action Specify the action to perform when a search meets the predicate condition.
Place search in a Pool
(the default) assigns searches that meet the predicate condition to the specified workload pool.
Abort search
kills the search process.
Display a Message
shows a message in the job inspector to users that have all of the following required capabilities:list_workload_pools
,edit_workload_pools
,list_workload_rules
, andselect_workload_rules
.
Move search to alternate Pool
moves the running search to a different specified pool.
Abort search
,Display a Message
, andMove search to alternate Pool
actions apply to in-progress searches only. You must specify aruntime
condition to enable these actions. For example, the predicateindex=_internal AND runtime>1m
triggers the specified action on all searches that containindex=_internal
and run for more than one minute.
ThePlace search in a pool
action is not valid with rules containing aruntime
condition.Abort search
,Display a Message
, andMove search to alternate Pool
actions are valid only when aruntime
condition exists.Workload Pool Select the workload pool to which this rule applies. User Message Enter a custom message that notifies the end user when a search triggers the workload rule action. For example, "Search runtime exceeded 30 seconds. The search was moved to the high_perf pool."
A user message is required with theDisplay a Message
action, and is optional for other actions.
When a search triggers the rule action, the user message appears in the Jobs manager in Splunk Web: Click Activity > Jobs > Job. It also appears under the Job menu in the Search app. - Click Submit.
Specify predicate values
The following table shows valid values for each workload rule predicate type:
Predicate type | Valid values |
---|---|
app |
Name of the app. For example, app=search The correct name to specify for an app is the name of the app directory located in |
role |
Name of the role. For example, role=admin .The For information on how role inheritance impacts |
user |
Name of any valid user. For example, user=bob . The reserved internal user "nobody" is invalid; the reserved internal user "splunk-system-user" is valid.The |
index |
Name of the index. For example, index=_internal . Value can refer to internal or public index.The
|
search_type |
adhoc , scheduled , datamodel_acceleration , report_acceleration , and summary_index
|
search_mode |
realtime and historical
|
search_time_range |
Supports alltime time range only.
|
runtime |
The amount of time that a search must run in a workload pool to trigger a specified action, such as Abort search , Display in Messages , or Move search to alternate Pool . For example, if you specify runtime>1m , any search in the pool that runs for more than 1 minute is subject to the specified action. By default, Valid units for |
Searches run by a single user can match multiple roles
By default, the admin
role in Splunk Enterprise inherits the user
role and the power
role. As a result of this role inheritance, in workload management, searches run by a single user can match to multiple roles in workload rules. For example, searches run by the "admin" user can match to workload rules that specify role=admin
, role=user
and role=power
.
To avoid having a single user, such as the "admin" user, match to multiple roles in your workload rules, specify user=<username>
in the rule predicate as an alternative to specifying role
.
For more information on role inheritance, see About configuring role-based user access in Securing Splunk Enterprise.
For more information on workload rule configuration settings, see workload_rules.conf in the Admin Manual.
Create a workload rule using the CLI
To create a workload rule using the CLI, run the splunk add workload-rule
command as follows, where predicate
has the syntax <type>=<value> with optional AND, NOT, OR, ():
./splunk add workload-rule <rule_name> -predicate <predicate> -workload_pool <pool_name>
For example:
./splunk add workload-rule "my_role_rule" -predicate "app=search AND (NOT index=_internal)" \ -workload_pool "pool_a"
To create a workload monitoring rule, the -predicate
value must include a runtime
condition, and specify the -action
value. For the "move" action, you must also specify the destination -workload_pool
value. For example:
./splunk add workload-rule "my_monitoring_rule" \ -predicate "(role!=admin AND (search_time_range=alltime OR index=*)) AND runtime>10m" \ -action <abort|move> -workload_pool "pool_b"
It is a best practice to use parentheses to group predicate conditions for clarity.
Create a workload rule using REST
Send a POST request to the following endpoint:
workloads/rules
For endpoint details, see workloads/rules in the REST API Reference Manual.
Enable workload rules
For instructions on how to enable and disable workload rules, see Enable individual workload rules.
View workload_rules.conf
When you create a workload rule using Splunk Web, the CLI, or REST, the rule configuration is stored in $SPLUNK_HOME/etc/apps/<app_name>/local/workload_rules.conf
. The default app context is search
.
It is not advisable to manually edit workload_rules.conf
. If a manual edit of workload_rules.conf
does occur, you must reload the file. Restart is not required.
workload_rules.conf
contains workload rule definitions that determine the order in which rules are evaluated, the mapping of rules to workload pools, the schedule for rules, and workload monitoring rules and actions. For example:
[workload_rules_order] rules = test_placement,test_abort,test_move [workload_rule:test_placement] predicate = app="search" AND (NOT index="_internal") workload_pool = low_priority_pool disabled = 0 [workload_rule:test_abort] action = abort predicate = index=* AND runtime>5seconds user_message = This search contains a wildcards and has exceeded the maximum runtime. disabled = 0 [workload_rule:test_move] action = move predicate = index=_internal AND runtime>5s user_message = This search has been moved to an alternate workload pool. workload_pool = low_priority_pool disabled = 0 [search_filter_rule:admission_rule_1] action = filter predicate = index=* AND search_time_range=alltime user_message = This search meets the predicate condition of admission_rule_1. The search cannot be executed. disabled = 1
For more information on workload rule settings, see workload_rules.conf.
Monitor triggered workload rule actions in the job inspector
When a running search triggers a workload rule action, information about the action appears in the Search job inspector. This includes the action that was taken on the search and the timestamp. If a search triggers multiple rules, the information appears in reverse chronological order.
To view details of a workload rule action:
- In Splunk Web, click Activity > Jobs.
- Find the specific search job and click Job > Inspect Job > Search job properties.
- View the details of the workload rule action under the
workload_action_information
property.
For more information on monitoring workload rule actions, see Monitor workload management using the Monitoring console.
Configure workload pools | Enable workload management |
This documentation applies to the following versions of Splunk® Enterprise: 9.0.0, 9.0.1, 9.0.2, 9.0.3, 9.0.4, 9.0.5, 9.0.6, 9.0.7, 9.0.8, 9.0.9, 9.0.10, 9.1.0, 9.1.1, 9.1.2, 9.1.3, 9.1.4, 9.1.5
Feedback submitted, thanks!