Configure workload pools

Before you can configure workload pools in Splunk Enterprise, you must set up Linux cgroups on your underlying Linux operating system. For instructions, see Set up Linux for workload management.

This topic discusses how to configure workload pools on a single instance. For information on how to extend this to distributed distributed deployments, see Configure workload management on distributed deployments.

Follow these steps to configure workload management on a single instance:

1. Run preflight checks.
2. Configure workload categories.
3. Create workload pools.
4. Configure workload rules.
5. Enable workload management.
6. Check workload management status.

Run preflight checks

When you open workload management in Splunk Web, a set of system checks runs automatically to determine if your underlying Linux operating system is set up properly for workload management.

You can optionally run preflight checks manually using the CLI or REST API.

If all preflight checks pass, this means your system is set up correctly and you can proceed to configure workload management. If any preflight checks fail, review the error messages to identify the Linux configuration issues you must fix before you can configure workload management.

Workload management preflight checks reflect the status of the local instance only.

Workload management runs the following preflight checks:

Name	Mitigation
Operating system	Operating system must be Linux. Workload management is not currently supported on Windows OS.
Cgroup version	Cgroup must be version 1. Workload management does not support pre-cgroup or cgroup version 2 Linux kernels.
CPU Splunk base directory check	CPU Splunk base directory Splunkd.service is missing. For systemd, the base directory is /sys/fs/cgroup/cpu/system.slice/<unit_file_name>. The <unit_file_name> is <SPLUNK_SERVER_NAME>.service. The <SPLUNK_SERVER_NAME> must match the Splunk server name in splunk-launch.conf. The default value is Splunkd. See Configure Linux systemd for workload management. For non-systemd, the base directory is /sys/fs/cgroup/cpu/splunk. The base directory name must match the workload_pool_base_dir_name defined in workload_pools.conf The default value is splunk. See Configure Linux systems not running systemd for workload management.
CPU Splunk base directory permissions	CPU Splunk base directory Splunkd.service requires read and write permissions. For systemd, permissions must be set for non-root user in the Splunkd.service unit file. See Configure Linux systemd for workload management. For non-systemd, use chown to grant permissions to the Splunk base directory. See Configure Linux systems not running systemd for workload managment.
Memory Splunk base directory check	Memory Splunk base directory Splunkd.service is missing. For systemd, the base directory is /sys/fs/cgroup/memory/system.slice/<unit_file_name>. See Configure Linux systemd for workload management. For non-systemd, the base directory is /sys/fs/cgroup/memory/splunk. See Configure Linux systems not running systemd for workload management.
Memory Splunk base directory permissions	Memory Splunk base directory Splunkd.service requires read and write permissions. For systemd, permissions must be set for non-root user in the Splunkd.service unit file. See Configure Linux systemd for workload management. For non-systemd, use chown to grant permissions to the Splunk base directory. See Configure Linux systems not running systemd for workload management.
Unit file check	The unit file Splunkd.service is missing. The unit file is located under /etc/systemd/system with the name <SPLUNK_SERVER_NAME>.service. SPLUNK_SERVER_NAME is set in splunk-launch.conf. See Configure Linux systemd for workload management.
Delegate property set to true	The Delegate property in the unit file must be set to true.
Splunk launched under systemd	splunkd is running as a systemd service. In the unit file, the Restart property must be set to always. The ExecStart property must include _internal_launch_under_systemd.

For more information on unit file properties, see systemd unit file properties.

For more information on how to set up Linux for workload management, see Set up Linux for workload management.

Run preflight checks in Splunk Web

1. Click Settings > Workload Management.
   The Linux preflight checks run automatically. If all preflight checks pass, the workload management UI opens, and you can proceed to configure workload management.
2. If any preflight check fails, a page appears showing the check results. Review the error messages and fix the specified Linux configuration issues.
   
3. After fixing the issues, click Rerun preflight checks.

Run preflight checks using the CLI

To run preflight checks for workload management using the CLI:

1. Log in to your Linux machine.
2. Run the following CLI command:

   ./splunk check workload-config
   

   Here is an example of the output from this command:

   Workload Management Preflight Checks failed. Fix the following issues:
   	CPU Splunk base directory Splunkd.service requires read and write permissions.
   	CPU Splunk base directory Splunkd.service is missing.
   	The 'Delegate' property in the unit file must be set to 'true'. Restart Splunk then rerun preflight checks.
   	In the unit file, the 'Restart' property must be set to 'always'. The 'ExecStart' property must include '_internal_launch_under_systemd'. Make sure the up-to-date unit file is loaded.
   	Memory Splunk base directory Splunkd.service requires read and write permissions.
   	Memory Splunk base directory Splunkd.service is missing.
   	Unit file Splunkd.service is missing. Restart Splunk then rerun preflight checks.
   

Run preflight checks using REST

Send a GET request to:

workloads/config/preflight-checks

For endpoint details, see workloads/config/preflight-checks in the REST API Reference Manual.

Configure workload categories

Workload categories determine the total amount of system CPU and memory resources available for workload pools running specific process types in Splunk Enterprise. For example, the search category determines the total amount of resources available to all workload pools running search processes. When you create a workload pool you must assign it to a workload category.

Workload management provides the following three workload categories:

• search: Scheduled searches and ad hoc searches, accelerated reports and data models.
• ingest: Indexing and other splunkd processes, including process runner, KV store, app server, and introspection.
• misc: Scripted inputs and modular inputs only.

Each workload category has its own CPU and memory resource allocation. And each workload pool within a category is assigned a fraction of the total CPU and a percentage of the total memory allocated to that category. You can modify the resource allocation for a category to ensure that sufficient resources are available to workload pools running high-priority processes.

The types of processes that run in each category are pre-defined and cannot be changed. For example, all search processes run in the search category, while other processes, such as the KV store process, run in the ingest category.

You can edit workload categories using Splunk Web, the CLI, or REST.

Edit workload categories using Splunk Web

To edit the resource allocation for a workload category in Splunk Web, do the following:

1. In Splunk Web, click Settings > Workload Management.
   The workload management UI opens.
2. Click the All Categories tile.
3. Click Edit under the specific category.
4. Specify the resource allocation:

   Field	Action
   CPU Weight	Specify the total CPU weight available for pools in this category. Unused CPU cycles are automatically shared with workloads in other categories.
   Memory Limit %	Specify the maximum percentage of Memory available for pools in this category.

5. Click Submit.

The percentage of CPU allocated to a category is a ratio of the total CPU weight across all categories. When you change the CPU weight for one category the CPU allocated to all other categories and all workload pools updates to reflect the change.

For more information, see Resource allocation in workload management.

Edit workload categories using the CLI

To edit the resource allocation for a workload category, run the following CLI command:

./splunk edit workload-category <category> [-cpu_weight <number> -mem_weight <number>]

where <category> is search, ingest, or misc.

To list workload category information, run the following CLI command:

./splunk list workload-category 

Edit workload categories using REST

To edit the resource allocation for a workload category, send a POST request to the following endpoint:

workloads/categories

For endpoint details, see workloads/categories in the REST API Reference Manual.

Create workload pools

A workload pool is a specified amount of CPU and memory resources that you define and allocate to processes in Splunk Enterprise.

To configure workload management, you must create, at a minimum, these two workload pools:

• Default pool in the search category: Searches that are not explicitly mapped to any workload rule are assigned to this pool by default.
• Default pool in the ingest category: Indexing and other non-search processes are assigned to this pool by default.

You can optionally create a default pool in the misc category. Scripted and modular inputs run in this pool by default. If you do not create a default pool in the misc category, scripted and modular inputs run in the default pool in the ingest category.

You can only create one workload pool in the ingest and misc categories.

You can create and edit workload pools using Splunk Web, the CLI, or REST API. Workload pool configurations are stored in workload_pools.conf. See View workload_pools.conf.

Create a workload pool in Splunk Web

1. In Splunk Web, click Settings > Workload Management.
2. Click Add Workload Pool.
3. Specify the following fields:

   Field	Action
   Pool Category	Select a workload category based on the type of process the pool will run (search, ingest, or misc). See Configure workload categories.
   Name	Specify the name of the workload pool. Valid characters are alphanumeric and underscore only.
   CPU Weight	The fraction of total available CPU for this pool. Unused CPU cycles are automatically shared with workloads in other pools.
   Memory Limit %	The maximum percentage of total available memory for this pool.
   Default Pool	Toggle the switch to make this pool the default pool for the selected category.

4. Click Submit.
   The workload pool appears in the Workload Management UI as shown in the following image:
   

For more information, see Resource allocation in workload management.

Create a workload pool using the CLI

Run the following CLI command:

./splunk add workload-pool <pool_name> [-category <search/ingest/misc> -cpu_weight <number> -mem_weight <number> -default_category_pool <true|false>]

Create a workload pool using REST

Send a POST request to the following endpoint:

workloads/pools 

For endpoint details, see workloads/pools in the REST API Reference Manual.

View workload_pools.conf

When you create a workload pool, the configuration is stored in $SPLUNK_HOME/etc/apps/<app_name>/local/workload_pools.conf.

workload_pools.conf defines the cpu and memory resource allocation for workload categories (search, ingest, and misc) and the individual workload pools created under those categories. For example:

[general]
default_pool = pool_1
ingest_pool = pool_3
enabled = 0
 
[workload_category:search]
cpu_weight = 70
mem_weight = 70
 
[workload_category:ingest]
cpu_weight = 20
mem_weight = 20
 
[workload_category:misc]
cpu_weight = 10
mem_weight = 10
 
[workload_pool:pool_1]
cpu_weight = 70
mem_weight = 70
category   = search
default_category_pool = 1
 
[workload_pool:pool_2]
cpu_weight = 30
mem_weight = 30
category   = search
default_category_pool = 0

[workload_pool:pool_3]
cpu_weight = 100
mem_weight = 100
category = ingest
default_category_pool = 1
 
[workload_pool:pool_4]
cpu_weight = 100
mem_weight = 100
category = misc
default_category_pool = 1

For more information, see workload_pools.conf.

Do not place workload_pools.conf files in more than one app context. Having identical workload_pools.conf stanzas in multiple app contexts can cause workload management enable/disable functions to fail and cause other issues.

Delete workload pools

You can delete any workload pool under a category, except for the default category pool. If you try to delete the default category pool an error message appears. You can delete workload pools using Splunk Web, the CLI, or REST API.

To delete a workload pool using the CLI:

./splunk remove workload-pool <pool_name>

You cannot delete a workload pool while a process is running in that pool. Any pool you delete that has an active process running in it will not be deleted until after workload_pools.conf reloads or Splunk Enterprise restarts.