How workload management works
This page provides an overview of workload management features and concepts.
Workload placement and monitoring
Workload management lets you reserve groups of CPU and memory resources on Linux operating systems, and allocate those resources to searches and other processes in Splunk Enterprise. These resource groups are called workload pools. You can create multiple workload pools of varying sizes to support the search requirements of different users and groups.
Workload management also lets you define policies called workload rules that determine which searches have access to specific workload pools. You can use workload rules to ensure that high-priority searches have access to sufficient resources, while less important searches are appropriately restricted.
You can create workload rules to assign individual searches to designated workload pools automatically, or assign searches to pools manually. For more information, see Manually assign searches to workload pools.
You can monitor your workload management configuration, and track CPU and memory usage on a per pool basis, using the workload management dashboards in the Monitoring Console. For more information, see Monitor workload management.
Resource allocation in workload management
CPU and memory resources are allocated in workload management in a resource pool hierarchy that includes 3 levels:
|Level 1: Top Splunk pool||The total amount of system CPU and memory resources allocated to workload management in Splunk Enterprise based on the cpu and memory resource allocation that you define in Linux cgroups. See cgroups.|
|Level 2: Workload categories||The amount of CPU and memory resources allocated to separate workload categories (search, ingest, and misc). See workload categories.|
|Level 3: Workload pools||The amount of CPU and memory resources allocated to individual workload pools under their respective categories. See workload pools.|
The following diagram illustrates the workload management resource pool hierarchy:
How CPU resource allocation works
Workload management calculates the amount of CPU allocated to a workload category as a ratio of the total CPU weight across all workload categories. If you change the CPU weight allocated to one category, the CPU percentage allocated to all other categories and their workload pools automatically updates to reflect the change.
The CPU weight allocated to a category determines the total amount of CPU resources available for all pools under that category. When you allocate CPU resources to an individual workload pool, workload management calculates the amount of CPU available to that pool as a percentage of the total CPU weight allocated to the parent category.
For example, if the CPU weight allocated to the search category is 70 out of a total weight of 100 across all 3 categories, and the CPU weight allocated to search_pool_1 is 50 out of a total weight of 100 across all pools in the search category, then the allocated CPU% for search_pool_1 is 50% of 70, or 35%. For information on configuring workload pools, see Create workload pools.
If an individual workload pool exceeds its allocated CPU %, it is considered a soft limit, and that pool can borrow available CPU resources from other pools in the category.
Memory resource allocation and sharing
The amount of memory allocated to an individual workload category is defined in terms of a memory limit percentage. The memory limit percentage determines the maximum amount of the total memory available to all categories that an individual category can use.
When you allocate memory to an individual workload pool, workload management calculates the amount of memory available to the pool as an absolute percentage of the total memory allocated to the category.
If an individual workload pool exceeds its memory limit, it can share memory with other pools in the same category, up to the total amount of memory available for the category.
Search and ingest process isolation
Workload management provides separate search and ingest categories to ensure resource isolation between search processes and ingest processes. Search and ingest process isolation lets you allocate resources to any number of search pools without affecting the resource allocation to the ingest category. See workload categories.
Workload management features and concepts
The following features and concepts are useful to understand before you configure and use workload management.
Workload categories determine the total amount of cpu and memory resources available to workload pools running specific process types in Splunk Enterprise. When you create a workload pool, you must assign it to a category. There are three workload categories:
- search: Scheduled searches, ad hoc searches, accelerated reports, and accelerated data models.
- ingest: Indexing and other
splunkdprocesses, including process runner, KV store, app server, and introspection.
- misc: Scripted inputs and modular inputs only.
For detailed information on how to allocate CPU and memory resources to workload categories, see Configure workload categories.
A workload pool is a specified amount of CPU and memory resources that you can define and allocate to processes in Splunk Enterprise. Each workload pool reserves a subset of the total amount of CPU and memory available in the workload category to which it belongs. You can assign individual scheduled or ad-hoc searches to designated workload pools, based on policies that you define in workload rules.
For detailed information on how to create and use workload pools, see Create workload pools.
A workload rule is a policy that you define to manage your search workloads. For example, you can create a rule that places a search in a specific workload pool, or create a rule that moves a long-running search to an alternate pool. There are two types of workload rules:
- Search placement rules
- Monitoring rules and actions
Search placement rules determine whether a search can be placed in a specified pool based on a user-defined condition. You can use search placement rules to help ensure that critical searches have access to sufficient resources, while less important searches are appropriately restricted.
Monitoring rules automatically trigger actions on a running search process based on a user-defined condition and the status of the search. Actions include abort, move to an alternate pool, and a message in the UI. Monitoring rules are useful for managing heavy search loads and preventing rogue processes from monopolizing pool resources.
You can also specify the order in which workload rules are evaluated to ensure that searches matching certain predicate conditions have priority access to specified workload pools.
For detailed information on how to define and use workload rules, see Create workload rules.
cgroups (control groups) are a Linux kernel feature that lets you prioritize a specified amount of system resources for a group of processes. cgroups also include a rules engine that lets you control user access to resources. Workload management in Splunk Enterprise is an abstraction of the underlying functionality of Linux cgroups.
Before you can configure workload management in Splunk Enterprise, you must set up cgroups on your Linux operating system. For more information, see Set up Linux for workload management.
systemd is a system startup and service manager for Linux operating systems that organizes processes under cgroups.
systemd uses instructions for a daemon specified in a unit configuration file. You can configure this unit file to run
splunkd as a
For information on how to configure
systemd for workload management, see Configure Linux systemd for workload management.
About workload management
This documentation applies to the following versions of Splunk® Enterprise: 8.0.0, 8.0.1, 8.0.2, 8.0.3