Configure workload management

Before you can configure workload management in Splunk Enterprise, you must set up Linux cgroups on your underlying Linux operating system. For instructions, see Set up Linux for workload management.

You can configure workload management on a single instance. For information on how to configure workload management on distributed deployments, see Configure workload management on distributed deployments.

Before you can enable workload management, you must create a default search pool and a default ingest pool. You can optionally create workload rules to control access to workload pools at any time. You can configure workload management using Splunk Web, CLI, or REST.

Follow these steps to configure workload management on a single instance:

1. Run preflight checks.
2. Create workload pools.
3. Create workload rules.
4. Enable workload management.
5. Check workload management status.

Run preflight checks

When you open workload management in Splunk Web, a set of preflight checks run automatically to determine if your underlying Linux operating system is set up properly for workload management.

If all preflight checks pass, then your system is set up correctly and you can configure workload management. If any preflight checks fail, review the error messages to identify the Linux configuration issues you must fix before you can configure workload management.

You can optionally run preflight checks manually using the CLI or REST.

Workload management preflight checks reflect the status of the local instance only.

Workload management runs the following preflight checks:

Name	Mitigation
Operating system	Operating system must be Linux. Workload management is not currently supported on Windows OS.
Cgroup Version	Cgroup must be version 1. Workload management does not support pre-cgroup or cgroup version 2 Linux kernels.
CPU Splunk base directory present	CPU Splunk base directory Splunkd.service is missing. For systemd, the base directory is /sys/fs/cgroup/cpu/system.slice/<unit_file_name>. The unit_file_name is <splunk_server_name>.service. The splunk_server_name must match the Splunk server name in splunk-launch.conf. The default value is Splunkd. See Configure systemd distributions for workload management. For non-systemd, the base directory is /sys/fs/cgroup/cpu/splunk. The base directory name must match the workload_pool_base_dir_name defined in workload_pools.conf The default value is splunk. See Configure non-systemd distributions.
CPU Splunk base directory permissions	CPU Splunk base directory Splunkd.service requires read and write permissions. For systemd, permissions must be set for non-root user in the Splunkd.service unit file. See Configure systemd distributions for workload management. For non-systemd, use chown to grant permissions to the splunk base directory. See Configure non-systemd distributions.
Memory Splunk base directory present	Memory Splunk base directory Splunkd.service is missing. For systemd, the base directory is /sys/fs/cgroup/memory/system.slice/<unit_file_name>. See Configure systemd distributions for workload management. For non-systemd, the base directory is /sys/fs/cgroup/memory/splunk. See Configure non-systemd distributions.
Memory Splunk base directory permissions	Memory Splunk base directory Splunkd.service requires read and write permissions. For systemd, permissions must be set for non-root user in the Splunkd.service unit file. See Configure systemd distributions for workload management. For non-systemd, use chown to grant permissions to the splunk base directory. See Configure non-systemd distributions.
Unit file present	The unit file Splunkd.service is missing. The unit file is located under /etc/systemd/system with the name <splunk_server_name>.service. splunk_server_name is set in splunk-launch.conf. See Configure systemd distributions for workload management.
Delegate property set to true	The Delegate property in the unit file must be set to true.
Splunk launched under systemd	splunkd is running as a systemd service. In the unit file, the Restart property must be set to always. The ExecStart property must include _internal_launch_under_systemd.

For more information on unit file properties, see systemd unit file properties.

For more information on how to set up Linux for workload management, see Set up Linux for workload management.

Run preflight checks in Splunk Web

1. Click Setttings > Workload Management.
   The Linux preflight checks run automatically. If all preflight checks pass, the workload management UI opens, and you can start to configure workload management.
2. If any preflight check fails, a page appears showing the check results. Review the error messages and fix the specified Linux configuration issues.
   
3. Click Rerun preflight checks.

Run preflight checks using the CLI

To run preflight checks for workload management using the CLI:

1. Log into your Linux machine.
2. Run the following CLI command.

   ./splunk check workload-config
   

   Here is an example of the output from this command:

   Workload Management Preflight Checks failed. Fix the following issues:
   	CPU Splunk base directory Splunkd.service requires read and write permissions.
   	CPU Splunk base directory Splunkd.service is missing.
   	The 'Delegate' property in the unit file must be set to 'true'. Restart Splunk then rerun preflight checks.
   	In the unit file, the 'Restart' property must be set to 'always'. The 'ExecStart' property must include '_internal_launch_under_systemd'. Make sure the up-to-date unit file is loaded.
   	Memory Splunk base directory Splunkd.service requires read and write permissions.
   	Memory Splunk base directory Splunkd.service is missing.
   	Unit file Splunkd.service is missing. Restart Splunk then rerun preflight checks.
   

Run preflight checks using REST

Send a GET request to:

workloads/config/preflight-checks

For endpoint details, see workloads/config/preflight-checks in the REST API Reference Manual.

Create workload pools

A workload pool is a specified amount of CPU and memory resources that you can define and allocate to search processes.

To configure workload management, you must create, at a minimum, these two workload pools:

Default search pool

Searches that are not explicitly mapped to a workload rule are assigned to this pool by default.

Default ingest pool

Indexing and other non-search processes are assigned to this pool by default.

You can specify if a pool is a default search pool or default ingest pool when you create the workload pool.

Create a workload pool in Splunk Web

1. In Splunk Web, click Settings > Workload Management.
2. Click Add Workload Pool.
3. Configure your new workload pool by defining the following fields:

   Field	Action
   Name	Specify the name of the workload pool. Valid characters are alphanumeric and underscore only.
   CPU %	Specify the percentage of the total CPU weight for the cpu control group.
   Memory %	Specify the percentage of the total memory weight for the memory control group.
   Default Search Pool	Toggle the switch to make this pool the default search pool.
   Default Ingest Pool	Toggle the switch to make this pool the default ingest pool.

   An individual workload pool cannot be both a default search pool and default ingest pool.

4. Click Submit.
   The new workload pool appears in the Workload Management UI.
   

Create a workload pool using the CLI

Run the following CLI command:

./splunk add workload-pool <pool_name> [-cpu_weight <group_name> -mem_weight <group_name> -default_pool <true|false> -ingest_pool <true|false>]

Create a workload pool using REST

Send a POST request to:

workloads/pools 

For endpoint details, see workloads/pools in the REST API Reference Manual.

View workload_pools.conf

When you create a workload pool, the configuration is stored in $SPLUNK_HOME/etc/apps/<app_name>/local/workload_pools.conf. For example:

[general]
default_pool = pool_1
ingest_pool = pool_2
enabled = 0
[workload_pool:pool_1]
cpu_weight = 20
mem_weight = 40
 
[workload_pool:pool_2]
cpu_weight = 80
mem_weight = 80

For more information on workload pool settings, see workload_pools.conf.

Create workload rules

Workload rules provide a policy-based method for assigning searches to workload pools. Each rule specifies a predicate condition that must match before you can assign searches to the designated pool. You can use workload rules to ensure that high-priority searches have access to adequate resources while low-priority searches are restricted.

Workload rules are evaluated in the order that you create them. If the predicate condition defined in a rule does not match, the next rule in order is evaluated. If there is no match with any rule, the search is assigned to the default search pool. In this way, workload rules let you prioritize the assignment of system resources based on conditions that you define.

You can only specify a single predicate for each workload rule.

Create a workload rule in Splunk Web

1. In Splunk Web, click Settings > Workload Management.
2. Click Add Workload Rule.
3. Configure your new workload rule by defining the following fields:

   Field	Action
   Name	Specify the name of the workload rule.
   Predicate	Specify a single predicate condition to access the workload pool. Enter as <type>=<value>, where valid <type> is "app" or "role". For example, a workload rule with predicate "app=itsi" maps all searches belonging to the ITSI app to the corresponding workload pool. Similarly, "role=admin" maps all searches for the admin role to the corresponding workload pool.
   Workload Pool	Select the workload pool to which this rule applies.

4. Click Submit.

Create a workload rule using the CLI

Run the following CLI command:

./splunk add workload-rule <rule_name> -predicate <type> -workload_pool <pool>

Create a workload rule using REST

Send a POST request to:

workloads/rules 

For endpoint details, see workloads/rules in the REST API Reference Manual.

View workload_rules.conf

When you create a workload rule, the configuration is stored in $SPLUNK_HOME/etc/apps/<app_name>/local/workload_rules.conf.

workload_rules.conf defines both the mappings to workload pools and the order in which rules are evaluated. For example:

[workload_rules_order]
rules = my_analyst_rule, my_app_rule
rules_number = 2

[workload_rule:my_app_rule]
predicate = app=search
workload_pool = my_app_pool

[workload_rule:my_analyst_rule]
predicate = role=analyst
workload_pool = my_analyst_pool

For more information on workload rule settings, see workload_rules.conf.

Enable or disable workload management

After you create your workload pools and rules, you must enable workload management. When you initiate a request to enable workload management, a series of health checks run in the background to validate both the workload management configuration and the underlying Linux system configuration. If these health checks fail, you cannot enable workload management and a failure message appears.

For more information on Linux configuration requirements, see Set up Linux for workload management.

Enable or disable workload management in Splunk Web

1. In Splunk Web, click Settings > Workload Management.
2. Toggle the switch to Enabled.
   This applies any pending configuration changes and enables workload management.

   To disable workload management, toggle the switch to '''Disabled'''.

Enable or disable workload management using the CLI

To enable or disable workload management, run the following CLI command:

./splunk <enable|disable> workload-management

Enable or disable workload management using REST

You can enable or disable workload management using REST. For endpoint details, see workloads/config/enable or workloads/config/disable in the REST API Reference Manual

Check workload management status

You can view the current active configuration of workload management using the CLI or REST. Output shows configuration details of all workload pools and rules, and whether workload management is supported and enabled on the instance.

Check workload management status using the CLI

Run the following CLI command:

./splunk show workload-management-status

Here is an example of the output from the command:

	Workload Management Status:
		Enabled: 1
		Supported: 1
		Ingest Pool: pool_5
		Default Pool: pool_2
		Error:
		Pending: 1

	Workload Pools:
		pool_2:
			CPU Group: /sys/fs/cgroup/cpu/splunk/pool_2
			Memory Group: /sys/fs/cgroup/memory/splunk/pool_2
			 CPU Weight: 20.00
			 Memory Weight: 20.00

		pool_4:
			CPU Group: /sys/fs/cgroup/cpu/splunk/pool_4
			Memory Group: /sys/fs/cgroup/memory/splunk/pool_4
			 CPU Weight: 35.00
			 Memory Weight: 35.00

		pool_5:
			CPU Group: /sys/fs/cgroup/cpu/splunk/pool_5
			Memory Group: /sys/fs/cgroup/memory/splunk/pool_5
			 CPU Weight: 30.00
			 Memory Weight: 30.00

		pool_6:
			CPU Group: /sys/fs/cgroup/cpu/splunk/pool_6
			Memory Group: /sys/fs/cgroup/memory/splunk/pool_6
			 CPU Weight: 15.00
			 Memory Weight: 15.00

	Workload Rules:
		rule_2:
			Order: 1
			Predicate : role=super_user
			Workload Pool: pool_2

		rule_3:
			Order: 2
			Predicate : role=analyst
			Workload Pool: pool_2

		rule_5:
			Order: 3
			Predicate : app=splunk_instrumentation
			Workload Pool: pool_4

Check workload management status using REST

To view workload management status information, send a GET request to:

workloads/status

For endpoint details, see workloads/status in the REST API Reference Manual.

Next Step

After you configure workload management, you can allocate resources to individual scheduled and ad-hoc search processes in Splunk Enterprise. For more information, see Assign searches to workload pools.