How workload management works

This page provides an overview of workload management features and concepts.

The basics

Workload management lets you reserve groups of CPU and memory resources on Linux operating systems, and allocate those resources to searches and other processes in Splunk Enterprise. These resource groups are called workload pools. You can create multiple workload pools of varying sizes to support the search requirements of different users and groups.

Workload management also lets you define policies called workload rules that determine which searches have access to specific workload pools. You can use workload rules to ensure that high-priority searches have access to sufficient resources, while less important searches are appropriately restricted.

You can create workload rules to assign individual searches to designated workload pools automatically, or assign searches to pools manually. For more information, see Manually assign searches to workload pools.

You can monitor your workload management configuration, and track CPU and memory usage on a per pool basis, using the workload management dashboards in the Monitoring Console. For more information, see Monitor workload management.

Resource allocation in workload management

CPU and memory resources are allocated in workload management in a resource pool hierarchy that includes 3 levels:

Level	Description
Level 1: Top Splunk pool	The total amount of system CPU and memory resources allocated to workload management in Splunk Enterprise based on the cpu and memory resource allocation that you define in Linux cgroups. See cgroups.
Level 2: Workload categories	The amount of CPU and memory resources allocated to separate workload categories (search, ingest, and misc). See workload categories.
Level 3: Workload pools	The amount of CPU and memory resources allocated to individual workload pools under their respective categories. See workload pools.

The following diagram illustrates the workload management resource pool hierarchy:

How CPU resource allocation works

Workload management calculates the amount of CPU allocated to a workload category as a ratio of the total CPU weight across all workload categories. If you change the CPU weight allocated to one category, the CPU percentage allocated to all other categories and their workload pools automatically updates to reflect the change.

The CPU weight allocated to one category determines the total amount of CPU resources available for all pools under that category. When you allocate CPU resources to an individual workload pool, workload management calculates the amount of CPU available to that pool as a percentage of the total CPU weight allocated to the parent category.

For example, if the CPU weight allocated to the search category is 70 out of a total weight of 100 across all 3 categories, and the CPU weight allocated to search_pool_1 is 50 out of a total weight of 100 across all pools in the search category, then the allocated CPU% for search_pool_1 is 50% of 70, or 35.00%. For detailed information on configuring workload pools, see Create workload pools.

If an individual workload pool exceeds its allocated CPU %, it is considered a soft limit, and that pool can borrow available CPU resources from other pools in the category.

Memory resource allocation and sharing

The amount of memory allocated to a workload category is defined as the maximum percentage of total available memory across all categories.

When you allocate memory to an individual workload pool, workload management calculates the amount of memory available to the pool as an absolute percentage of the total memory allocated to the category.

If an individual workload pool exceeds its memory limit, it can share memory with other pools in the same category, up to the total amount of memory available for the category.

Search and ingest process isolation

Workload management provides separate search and ingest categories that ensure resource isolation between search processes and ingest processes. Search and ingest process isolation lets you allocate resources to any number of search pools without affecting the resource allocation for the ingest category. See workload categories.

Workload management features and concepts

The following features and concepts are useful to understand before you configure and use workload management.

Workload categories

Workload categories determine the total amount of cpu and memory resources available to workload pools running specific process types in Splunk Enterprise. When you create a workload pool, you must assign it to a category. There are three workload categories:

• search: Scheduled searches, ad hoc searches, accelerated reports, and accelerated data models.
• ingest: Indexing and other splunkd processes, including process runner, KV store, app server, and introspection.
• misc: Scripted inputs and modular inputs only.

For detailed information on how to allocate CPU and memory resources to workload categories, see Configure workload categories.

Workload pools

A workload pool is a specified amount of CPU and memory resources that you can define and allocate to processes in Splunk Enterprise. Each workload pool reserves a subset of the total amount of CPU and memory available in the workload category to which it belongs. You can assign individual scheduled or ad-hoc searches to designated workload pools, based on policies that you define in workload rules.

For detailed information on how to create and use workload pools, see Create workload pools.

Workload rules

A workload rule is a policy that you define to control access to workload pools. Each workload rule has a user-defined predicate (condition) that determines whether a search can access a designated pool. You can also specify an order for workload rules that determines which searches have priority access to a workload pool.

For detailed information on how to define and use workload rules, see Create workload rules.

cgroups

cgroups (control groups) are a Linux kernel feature that lets you prioritize a specified amount of system resources for a group of processes. cgroups also include a rules engine that lets you control user access to resources. Workload management in Splunk Enterprise is an abstraction of the underlying functionality of Linux cgroups.

Before you can configure workload management in Splunk Enterprise, you must set up cgroups on your Linux operating system. For more information, see Set up Linux for workload management.

systemd

systemd is a system startup and service manager for Linux operating systems that organizes processes under cgroups. systemd uses instructions for a daemon specified in a unit configuration file. You can configure this unit file to run splunkd as a systemd service.

For information on how to configure systemd for workload management, see Configure Linux systemd for workload management.