Configure workload rules

Workload rules provide an automated way to assign searches to workload pools and monitor running searches. Workload management evaluates workload rules in the order in which they are listed. If a search meets the predicate condition defined in a rule, a specified action is taken. Workload rules are evaluated for every new search and reevaluated every 10 seconds.

There are two types of workload rules:

• Search placement rules
• Search monitoring rules

Search placement rules determine the pool in which a search is placed when you start a search. Predicates that you can define to control search placement include app, role, user, index, search_type, search_mode, and search_time_range. You can use search placement rules to ensure that high-priority searches are assigned to pools that provide adequate resources, while low-priority searches are restricted.

Search monitoring rules automatically trigger actions on running searches based on the defined rule predicate and the status of the search. When you create a monitoring rule, you must specify a runtime value in the predicate. If a search exceeds the runtime value, workload management performs the specified action. Supported actions are Abort search, Display in Messages, and Move search to alternate Pool. You can use monitoring rules to manage heavy search loads and prevent rogue processes from monopolizing pool resources.

Workload rules apply to base searches only. Workload rules do not apply to subsearches.

Create a workload rule in Splunk Web

To create a workload rule in Splunk Web:

1. In Splunk Web, click Settings > Workload Management.
2. Click Add Workload Rule.
3. Define the following fields to configure a new workload rule:

   Field	Action
   Name	Specify the name of the workload rule.
   Predicate (Condition)	Specify a predicate (condition) that must match to trigger this rule. The predicate syntax is <type>=<value> with optional AND, OR, NOT, (). For example, app=search AND role=power triggers all searches belonging to both the Search app and the power role. Valid predicate types are app, role, index, user, search_type, search_mode, search_time_range, and runtime. For supported predicate values, see Specify predicate values in the next section on this page. In complex predicates, AND, OR, and NOT operators must be upper case. Lower case is not supported.
   Schedule	(Optional) Set a schedule for the workload rule. The schedule determines the time period during which the rule is valid. If set to Always On (the default), the rule remains valid indefinitely and does not expire. If set to Time Range, the rule is valid during the specified time range only and expires when the time range ends. If set to Every Day, Every Week, or Every Month, the rule becomes valid on a recurring basis during the specified time range every day, on the specified days of the week, or on the specified days of the month. The schedule time for a workload rule is based on the system timezone, regardless of the timezone set for an individual user in the UI.
   Action	Specify the action to perform when a search meets the predicate condition. Place search in a Pool (the default) assigns searches that meet the predicate condition to the specified workload pool. Abort search kills the search process. Display a Message shows a message in the job inspector to users that have all of the following required capabilities: list_workload_pools, edit_workload_pools, list_workload_rules, and select_workload_rules. Move search to alternate Pool moves the running search to a different specified pool. Abort search, Display a Message, and Move search to alternate Pool actions apply to in-progress searches only. You must specify a runtime condition to enable these actions. For example, the predicate index=_internal AND runtime>1m triggers the specified action on all searches that contain index=_internal and run for more than one minute. The Place search in a pool action is not valid with rules containing a runtime condition. Abort search, Display in Messages, and Move search to alternate Pool actions are valid only when a runtime condition exists.
   Workload Pool	Select the workload pool to which this rule applies.
   User Message	Enter a custom message that notifies the end user when a search triggers the workload rule action. For example, "Search runtime exceeded 30 seconds. The search was moved to the high_perf pool." A user message is required with the Display a Message action, and is optional for other actions. Messages are limited to a maximum of 140 characters. When a search triggers the rule action, the user message appears in the Jobs manager in Splunk Web: Click Activity > Jobs > Job. It also appears under the Job menu in the Search app.

4. Click Submit.

Specify predicate values

The table describes valid predicate values for each type of workload rule predicate:

Predicate type	Valid values
app	Name of the app. For example, app=search The correct name to specify for an app is the name of the app directory located in $SPLUNK_HOME/etc/apps. You can also find the correct name for an app in Splunk Web: Click Apps > Manage Apps. See app names listed under Folder name. App names are case insensitive.
role	Name of the role. For example, role=admin. The role predicate supports a wildcard (*) in the rule definition. For example, a rule that contains role=support_* will match the role names support_team1, support_team2 and so on. A rule that contains role=* will match all role name values. For information on how role inheritance impacts role predicate matches, see Searches run by single user can match multiple roles in the Workload Management manual in the Splunk Enterprise documentation.
user	Name of any valid user. For example, user=bob. The reserved internal user "nobody" is invalid. The reserved internal user "splunk-system-user" is valid. The user predicate supports a wildcard (*) in the rule definition. For example, a rule that contains user=* will match all user name values.
index	Name of the index. For example, index=_internal. Value can refer to internal or public index. The index predicate supports a wildcard (*) in the rule definition. For example, a rule that contains index=support_*, will match the index names support_prod, support_test, and so on. A rule that contains index=_* will match any index name that starts with an underscore, such as _internal, _audit, and _introspection. index=* is a special case, where * is not treated as a wildcard, but as a literal string match. For example, if a rule contains index=*, it will match the exact string index=* or index=_*. index=* will not match all internal indexes, such as index=_internal, index=_audit, and so on.
search_type	Supports adhoc, scheduled, datamodel_acceleration, report_acceleration, and summary_index search types.
search_mode	Supports realtime and historical search modes.
search_time_range	search_time_range predicate values match against the time range of a search. For example, if you specify search_time_range>4h, any search whose time range exceeds 4 hours is subject to the specified action. Supports alltime and numerical values with time units m, minute, minutes, h, hour, hours, w, weeks, d, day, or days. Supports comparison operators =, >, and <. search_time_range predicate values match against time ranges selected using the Time Range Picker in the UI and time ranges defined by earliest and latest time modifiers for historical searches specified in a search string. See Specify time modifiers in your search. Searches that run over the "all time" time range only match against rules that specify the alltime value, and do not match against rules that specify numerical values. For example, an all time search defined by earliest=0 and latest=now only matches rules that specify search_time_range=alltime, and does not match rules that specify a numerical value, such as search_time_range>10m. search-time_range predicate values that contain an equal sign (=) might not match against search time ranges that use "snap to" time units. For example, while earliest=-24h and latest=now defines a time range of 24 hours, when using "snap to" time units, such as earliest=-24h@h and latest=now, the time range can be evaluated as greater than or equal to 24 hours, depending on what time the search runs. See Relative time modifiers that snap to a time. To ensure search_time_range values match against time ranges with "snap to" time units, avoid using the = operator, and instead specify a value such as search_time_range>24h. srchTimeWin and srchTimeEarliest settings in authorize.conf, which let you set maximum time range and earliest event time for searches, respectively, do not apply to workload rules and admission rules. For example, if you run a search with a time range of 7 days, and you have an admission rule that specifies search_time_range>48h, the search will be filtered out, even if the srchTimeWin value for that role is set to 24 hours or greater.
runtime	The amount of time that a search must run in a workload pool to trigger a specified action, such as Abort search, Display in Messages, or Move search to alternate Pool. For example, if you specify runtime>1m, any search in the pool that runs for more than 1 minute is subject to the specified action. By default, runtime applies to both historical and real-time searches. If you want the runtime condition to apply to historical searches only or real-time searches only, you must specify the search_mode in the predicate condition. For example, if you specify runtime>5m AND search_mode=realtime, only those realtime searches in the pool that run for more than 5 minutes are subject to the specified action. Valid time units for runtime values include s, second, seconds, m, minute, minutes, and h, hour, hours. Note: runtime applies to workload rules only. It is not a valid predicate type for admission rules.

For workload rule use case examples, see Workload Management examples.

Enable workload rules

You can enable or disable individual workload rules. This lets you create and save multiple different workload rules and apply them as needed. Individual workload rules are enabled by default when you create them. Disabled workload rules are not evaluated and have no effect on running searches.

To enable or disable an individual workload rule:

1. In Splunk Web, click Settings > Workload Management > Workload Rules.
2. In the Status column, toggle the switch to enable or disable the individual workload rule.

The workload management feature must be enabled for workload rules to apply to searches. In Splunk Cloud Platform, the workload management feature is enabled for the sc_admin role by default and cannot be disabled.

Monitor triggered workload rule actions

When a running search triggers a workload rule action, information about the action appears in the Search job inspector. This includes the action that was taken on the search and the timestamp. If a search triggers multiple rules, the information appears in reverse chronological order.

To view details of a workload rule action:

1. In Splunk Web, click Activity > Jobs.
2. Find the specific search job and click Job > Inspect Job > Search job properties.
3. View details of the workload rule action under the workload_action_information property.
   

To view the workload_action_information property, you must have list_workload_pools and select_workload_pools capabilities.