Configure workload management

Before you can configure workload management in Splunk Enterprise, you must set up Linux cgroups on your underlying Linux operating system. For instructions, see Set up Linux for workload management.

You can configure workload management on a single instance. For information on how to configure workload management on distributed deployments, see Configure workload management on distributed deployments.

Before you can enable workload management, you must create a default search pool and a default ingest pool. You can optionally create workload rules to control access to workload pools at any time. You can configure workload management using Splunk Web, CLI, or REST.

Follow these steps to configure workload management on a single instance:

1. Create workload pools.
2. Create workload rules.
3. Enable workload management.
4. Check workload management status.

Create workload pools

A workload pool is a specified amount of CPU and memory resources that you can define and allocate to search processes.

To configure workload management, you must create, at a minimum, these two workload pools:

Default search pool

Searches that are not explicitly mapped to a workload rule are assigned to this pool by default.

Default ingest pool

Indexing and other non-search processes are assigned to this pool by default.

You can specify if a pool is a default search pool or default ingest pool when you create the workload pool.

Create a workload pool in Splunk Web

1. In Splunk Web, click Settings > Workload Management.
2. Click Add Workload Pool.
3. Configure your new workload pool by defining the following fields:

   Field	Action
   Name	Specify the name of the workload pool. Valid characters are alphanumeric and underscore only.
   CPU %	Specify the percentage of the total CPU weight for the cpu control group.
   Memory %	Specify the percentage of the total memory weight for the memory control group.
   Default Search Pool	Toggle the switch to make this pool the default search pool.
   Default Ingest Pool	Toggle the switch to make this pool the default ingest pool.

   An individual workload pool cannot be both a default search pool and default ingest pool.

4. Click Submit.
   The new workload pool appears in the Workload Management UI.
   

Create a workload pool using the CLI

Run the following CLI command:

./splunk add workload-pool <pool_name> [-cpu_weight <group_name> -mem_weight <group_name> -default_pool <true|false> -ingest_pool <true|false>]

Create a workload pool using REST

Send a POST request to:

workloads/pools 

For endpoint details, see workloads/pools in the REST API Reference Manual.

View workload_pools.conf

When you create a workload pool, the configuration is stored in $SPLUNK_HOME/etc/apps/<app_name>/local/workload_pools.conf.

[general]
default_pool = pool_1
ingest_pool = pool_2
enabled = 0
[workload_pool:pool_1]
cpu_weight = 20
mem_weight = 40
 
[workload_pool:pool_2]
cpu_weight = 80
mem_weight = 80

For more information on workload pool settings, see workload_pools.conf.

Create workload rules

Workload rules provide a policy-based method for assigning searches to workload pools. Each rule specifies a predicate condition that must match before you can assign searches to the designated pool. You can use workload rules to ensure that high-priority searches have access to adequate resources while low-priority searches are restricted.

Workload rules are evaluated in the order that you create them. If the predicate condition defined in a rule does not match, the next rule in order is evaluated. If there is no match with any rule, the search is assigned to the default search pool. In this way, workload rules let you prioritize the assignment of system resources based on conditions that you define.

You can only specify a single predicate for each workload rule.

Create a workload rule in Splunk Web

1. In Splunk Web, click Settings > Workload Management.
2. Click Add Workload Rule.
3. Configure your new workload rule by defining the following fields:

   Field	Action
   Name	Specify the name of the workload rule.
   Predicate	Specify a single predicate condition to access the workload pool. Enter as <type>=<value>, where valid <type> is "app" or "role". For example, a workload rule with predicate "app=itsi" maps all searches belonging to the ITSI app to the corresponding workload pool. Similarly, "role=admin" maps all searches for the admin role to the corresponding workload pool.
   Workload Pool	Select the workload pool to which this rule applies.

4. Click Submit.

Create a workload rule using the CLI

Run the following CLI command:

./splunk add workload-rule <rule_name> -predicate <type> -workload_pool <pool>

Create a workload rule using REST

Send a POST request to:

workloads/rules 

For endpoint details, see workloads/rules in the REST API Reference Manual.

View workload_rules.conf

When you create a workload rule, the configuration is stored in $SPLUNK_HOME/etc/apps/<app_name>/local/workload_rules.conf.

workload_rules.conf defines both the mappings to workload pools and the order in which rules are evaluated. For example:

[workload_rules_order]
rules = my_analyst_rule, my_app_rule
rules_number = 2

[workload_rule:my_app_rule]
predicate = app=search
workload_pool = my_app_pool

[workload_rule:my_analyst_rule]
predicate = role=analyst
workload_pool = my_analyst_pool

For more information on workload rule settings, see workload_rules.conf.

Enable or disable workload management

After you create your workload pools and rules, you must enable workload management. When you initiate a request to enable workload management, a series of health checks run in the background to validate both the workload management configuration and the underlying Linux system configuration. If these health checks fail, you cannot enable workload management and a failure message appears.

For more information on Linux configuration requirements, see Set up Linux for workload management.

Enable or disable workload management in Splunk Web

1. In Splunk Web, click Settings > Workload Management.
2. Toggle the switch to Enabled.
   This applies any pending configuration changes and enables workload management.

   To disable workload management, toggle the switch to '''Disabled'''.

Enable or disable workload management using the CLI

To enable or disable workload management, run the following CLI command:

./splunk <enable|disable> workload-management

Enable or disable workload management using REST

You can enable or disable workload management using REST. For endpoint details, see workloads/config/enable or workloads/config/disable in the REST API Reference Manual

Check workload management status

You can view the current active configuration of workload management using the CLI or REST. Output shows configuration details of all workload pools and rules, and whether workload management is supported and enabled on the instance.

Check workload management status using the CLI

Run the following CLI command:

./splunk show workload-management-status

Here is an example of the output from the command:

	Workload Management Status:
		Enabled: 1
		Supported: 1
		Ingest Pool: pool_5
		Default Pool: pool_2
		Error:
		Pending: 1

	Workload Pools:
		pool_2:
			CPU Group: /sys/fs/cgroup/cpu/splunk/pool_2
			Memory Group: /sys/fs/cgroup/memory/splunk/pool_2
			 CPU Weight: 20.00
			 Memory Weight: 20.00

		pool_4:
			CPU Group: /sys/fs/cgroup/cpu/splunk/pool_4
			Memory Group: /sys/fs/cgroup/memory/splunk/pool_4
			 CPU Weight: 35.00
			 Memory Weight: 35.00

		pool_5:
			CPU Group: /sys/fs/cgroup/cpu/splunk/pool_5
			Memory Group: /sys/fs/cgroup/memory/splunk/pool_5
			 CPU Weight: 30.00
			 Memory Weight: 30.00

		pool_6:
			CPU Group: /sys/fs/cgroup/cpu/splunk/pool_6
			Memory Group: /sys/fs/cgroup/memory/splunk/pool_6
			 CPU Weight: 15.00
			 Memory Weight: 15.00

	Workload Rules:
		rule_2:
			Order: 1
			Predicate : role=super_user
			Workload Pool: pool_2

		rule_3:
			Order: 2
			Predicate : role=analyst
			Workload Pool: pool_2

		rule_5:
			Order: 3
			Predicate : app=splunk_instrumentation
			Workload Pool: pool_4

Check workload management status using REST

To view workload management status information, send a GET request to:

workloads/status

For endpoint details, see workloads/status in the REST API Reference Manual.

Next Step

After you configure workload management, you can allocate resources to individual scheduled and ad-hoc search processes in Splunk Enterprise. For more information, see Assign searches to workload pools.