Splunk® Enterprise

Workload Management

Download manual as PDF

Download topic as PDF

Configure workload management

This topic discusses how to configure workload management on a single instance. For information on how to extend this configuration to distributed deployments, see Configure workload management on distributed deployments.

Prerequisites

Before you can configure workload management in Splunk Enterprise, you must set up cgroups on your Linux operating system. For instructions, see Set up Linux for workload management.

Configuration overview

You can configure workload management using Splunk Web, the command line interface (CLI), or REST API.

Before you can enable workload management, you must create a default workload pool in the search category and a default workload pool in the ingest category. Optionally, you can create workload rules to control access to workload pools.

Follow these steps to configure workload management on a single instance:

  1. Run preflight checks.
  2. Configure workload categories.
  3. Create workload pools.
  4. Create workload rules.
  5. Enable workload management.
  6. Check workload management status.

Run preflight checks

When you open workload management in Splunk Web, a set of system checks runs automatically to determine if your underlying Linux operating system is set up properly for workload management.

You can optionally run preflight checks manually using the CLI or REST API.

If all preflight checks pass, this means your system is set up correctly and you can proceed to configure workload management. If any preflight checks fail, review the error messages to identify the Linux configuration issues you must fix before you can configure workload management.

Workload management preflight checks reflect the status of the local instance only.

Workload management runs the following preflight checks:

Name Mitigation
Operating system Operating system must be Linux. Workload management is not currently supported on Windows OS.
Cgroup version Cgroup must be version 1. Workload management does not support pre-cgroup or cgroup version 2 Linux kernels.
CPU Splunk base directory check CPU Splunk base directory Splunkd.service is missing.


For systemd, the base directory is /sys/fs/cgroup/cpu/system.slice/<unit_file_name>.
The <unit_file_name> is <SPLUNK_SERVER_NAME>.service. The <SPLUNK_SERVER_NAME> must match the Splunk server name in splunk-launch.conf. The default value is Splunkd. See Configure Linux systemd for workload management.

For non-systemd, the base directory is /sys/fs/cgroup/cpu/splunk.
The base directory name must match the workload_pool_base_dir_name defined in workload_pools.conf The default value is splunk. See Configure Linux systems not running systemd for workload management.

CPU Splunk base directory permissions CPU Splunk base directory Splunkd.service requires read and write permissions.


For systemd, permissions must be set for non-root user in the Splunkd.service unit file.
See Configure Linux systemd for workload management.

For non-systemd, use chown to grant permissions to the Splunk base directory.
See Configure Linux systems not running systemd for workload managment.

Memory Splunk base directory check Memory Splunk base directory Splunkd.service is missing.


For systemd, the base directory is /sys/fs/cgroup/memory/system.slice/<unit_file_name>.
See Configure Linux systemd for workload management.

For non-systemd, the base directory is /sys/fs/cgroup/memory/splunk.
See Configure Linux systems not running systemd for workload management.

Memory Splunk base directory permissions Memory Splunk base directory Splunkd.service requires read and write permissions.


For systemd, permissions must be set for non-root user in the Splunkd.service unit file.
See Configure Linux systemd for workload management.

For non-systemd, use chown to grant permissions to the Splunk base directory.
See Configure Linux systems not running systemd for workload management.

Unit file check The unit file Splunkd.service is missing.


The unit file is located under /etc/systemd/system with the name <SPLUNK_SERVER_NAME>.service. SPLUNK_SERVER_NAME is set in splunk-launch.conf. See Configure Linux systemd for workload management.

Delegate property set to true The Delegate property in the unit file must be set to true.
Splunk launched under systemd splunkd is running as a systemd service. In the unit file, the Restart property must be set to always. The ExecStart property must include _internal_launch_under_systemd.

For more information on unit file properties, see systemd unit file properties.

For more information on how to set up Linux for workload management, see Set up Linux for workload management.

Run preflight checks in Splunk Web

  1. Click Settings > Workload Management.
    The Linux preflight checks run automatically. If all preflight checks pass, the workload management UI opens, and you can proceed to configure workload management.
  2. If any preflight check fails, a page appears showing the check results. Review the error messages and fix the specified Linux configuration issues.
    Wlm preflight checks 8.x.png
  3. After fixing the issues, click Rerun preflight checks.

Run preflight checks using the CLI

To run preflight checks for workload management using the CLI:

  1. Log in to your Linux machine.
  2. Run the following CLI command:
    ./splunk check workload-config
    

    Here is an example of the output from this command:

    Workload Management Preflight Checks failed. Fix the following issues:
    	CPU Splunk base directory Splunkd.service requires read and write permissions.
    	CPU Splunk base directory Splunkd.service is missing.
    	The 'Delegate' property in the unit file must be set to 'true'. Restart Splunk then rerun preflight checks.
    	In the unit file, the 'Restart' property must be set to 'always'. The 'ExecStart' property must include '_internal_launch_under_systemd'. Make sure the up-to-date unit file is loaded.
    	Memory Splunk base directory Splunkd.service requires read and write permissions.
    	Memory Splunk base directory Splunkd.service is missing.
    	Unit file Splunkd.service is missing. Restart Splunk then rerun preflight checks.
    

Run preflight checks using REST

Send a GET request to:

workloads/config/preflight-checks

For endpoint details, see workloads/config/preflight-checks in the REST API Reference Manual.

Configure workload categories

Workload categories determine the total amount of system CPU and memory resources available for workload pools running specific process types in Splunk Enterprise. For example, the search category determines the total amount of resources available to all workload pools running search processes. When you create a workload pool you must assign it to a workload category.

Workload management provides the following three workload categories:

  • search: Scheduled searches and ad hoc searches, accelerated reports and data models.
  • ingest: Indexing and other splunkd processes, including process runner, KV store, app server, and introspection.
  • misc: Scripted inputs and modular inputs only.

Each workload category has its own CPU and memory resource allocation. And each workload pool within a category is assigned a fraction of the total CPU and a percentage of the total memory allocated to that category. You can modify the resource allocation for a category to ensure that sufficient resources are available to workload pools running high-priority processes.

The types of processes that run in each category are pre-defined and cannot be changed. For example, all search processes run in the search category, while other processes, such as the KV store process, run in the ingest category.

You can edit workload categories using Splunk Web, the CLI, or REST.

Edit workload categories using Splunk Web

To edit the resource allocation for a workload category in Splunk Web, do the following:

  1. In Splunk Web, click Settings > Workload Management.
    The workload management UI opens.
  2. Click the All Categories tile.
  3. Click Edit under the specific category.
  4. Specify the resource allocation:
    Field Action
    CPU Weight Specify the total CPU weight available for pools in this category. Unused CPU cycles are automatically shared with workloads in other categories.
    Memory Limit % Specify the maximum percentage of Memory available for pools in this category.
  5. Click Submit.

The percentage of CPU allocated to a category is a ratio of the total CPU weight across all categories. When you change the CPU weight for one category the CPU allocated to all other categories and all workload pools updates to reflect the change.

For more information, see Resource allocation in workload management.

Edit workload categories using the CLI

To edit the resource allocation for a workload category, run the following CLI command:

./splunk edit workload-category <category> [-cpu_weight <number> -mem_weight <number>]

where <category> is search, ingest, or misc.

To list workload category information, run the following CLI command:

./splunk list workload-category 

Edit workload categories using REST

To edit the resource allocation for a workload category, send a POST request to the following endpoint:

workloads/categories

For endpoint details, see workloads/categories in the REST API Reference Manual.

Create workload pools

A workload pool is a specified amount of CPU and memory resources that you define and allocate to processes in Splunk Enterprise.

To configure workload management, you must create, at a minimum, these two workload pools:

  • Default pool in the search category: Searches that are not explicitly mapped to any workload rule are assigned to this pool by default.
  • Default pool in the ingest category: Indexing and other non-search processes are assigned to this pool by default.

You can optionally create a default pool in the misc category. Scripted and modular inputs run in this pool by default. If you do not create a default pool in the misc category, scripted and modular inputs run in the default pool in the ingest category.

You can only create one workload pool in the ingest and misc categories.

Create a workload pool in Splunk Web

  1. In Splunk Web, click Settings > Workload Management.
  2. Click Add Workload Pool.
  3. Specify the following fields:
    Field Action
    Pool Category Select a workload category based on the type of process the pool will run (search, ingest, or misc). See Configure workload categories.
    Name Specify the name of the workload pool. Valid characters are alphanumeric and underscore only.
    CPU Weight The fraction of total available CPU for this pool. Unused CPU cycles are automatically shared with workloads in other pools.
    Memory Limit % The maximum percentage of total available memory for this pool.
    Default Pool Toggle the switch to make this pool the default pool for the selected category.
  4. Click Submit.
    The workload pool appears in the Workload Management UI as shown in the following image:
    The image shows the Workload Management UI with the default search pool highlighted under the search category.

For more information, see Resource allocation in workload management.

Create a workload pool using the CLI

Run the following CLI command:

./splunk add workload-pool <pool_name> [-category <search/ingest/misc> -cpu_weight <number> -mem_weight <number> -default_category_pool <true|false>]

Create a workload pool using REST

Send a POST request to the following endpoint:

workloads/pools 

For endpoint details, see workloads/pools in the REST API Reference Manual.

View workload_pools.conf

When you create a workload pool, the configuration is stored in $SPLUNK_HOME/etc/apps/<app_name>/local/workload_pools.conf.

workload_pools.conf defines the cpu and memory resource allocation for workload categories (search, ingest, and misc) and the individual workload pools created under those categories. For example:

[general]
default_pool = pool_1
ingest_pool = pool_3
enabled = 0
 
[workload_category:search]
cpu_weight = 70
mem_weight = 70
 
[workload_category:ingest]
cpu_weight = 20
mem_weight = 20
 
[workload_category:misc]
cpu_weight = 10
mem_weight = 10
 
[workload_pool:pool_1]
cpu_weight = 70
mem_weight = 70
category   = search
default_category_pool = 1
 
[workload_pool:pool_2]
cpu_weight = 30
mem_weight = 30
category   = search
default_category_pool = 0

[workload_pool:pool_3]
cpu_weight = 100
mem_weight = 100
category = ingest
default_category_pool = 1
 
[workload_pool:pool_4]
cpu_weight = 100
mem_weight = 100
category = misc
default_category_pool = 1

For more information, see workload_pools.conf.

Do not place workload_pools.conf files in more than one app context. Having identical workload_pools.conf stanzas in multiple app contexts can cause workload management enable/disable functions to fail and cause other issues.

Delete workload pools

You can delete any workload pool under a category, except for the default category pool. If you try to delete the default category pool an error message appears. You can delete workload pools using Splunk Web, the CLI, or REST API.

To delete a workload pool using the CLI:

./splunk remove workload-pool <pool_name>

You cannot delete a workload pool while a process is running in that pool. Any pool you delete that has an active process running in it will not be deleted until after workload_pools.conf reloads or Splunk Enterprise restarts.

Create workload rules

Workload rules are policies that you define to manage your search workloads automatically. Each rule specifies a predicate condition that a search must meet before an action is taken, such as placing the search in a specified workload pool or moving the search to an alternate pool.

There are two types of workload rules:

  • Search placement rules
  • Monitoring rules and actions

Search placement rules determine the pool in which a search is placed when you start a search. Predicates you can define to control search placement include app, role, user, index, search_type, search_mode, and search_time_range. You can use search placement rules to ensure that high-priority searches are assigned to pools that provide adequate resources, while low-priority searches are restricted.

Monitoring rules automatically trigger actions on running search processes based on the rule predicate and the status of the search. When you create a monitoring rule, you must specify a runtime value in the predicate. If a search exceeds the runtime value, workload management performs the specified action. Supported actions are Abort search, Display in Messages, and Move search to alternate Pool. You can use monitoring rules to manage heavy search loads and prevent rogue processes from monopolizing pool resources.

Workload rules are evaluated in the order in which they appear in the rules list in the UI, top to bottom. If a search meets the conditions of a rule, the specified action is taken and no further rules are evaluated. You can edit the order of workload rules to ensure desired behavior. If a search does not meet the conditions of any rule, the search is automatically placed in the default search pool. Workload rules are evaluated for every new search and reevaluated every few seconds.


Create a workload rule in Splunk Web

  1. In Splunk Web, click Settings > Workload Management.
  2. Click Add Workload Rule.
  3. Define the following fields to configure a new workload rule:
    Field Action
    Name Specify the name of the workload rule.
    Predicate (Condition) Specify a predicate (condition) that must match for a search to access the specified workload pool. The predicate syntax is <type>=<value> with optional AND, OR, NOT, (). For example, app=search AND role=power maps all searches belonging to the Search app executed by a user that belongs to the power role to the corresponding workload pool.

    Valid predicate types are app, role, index, user, search_type, search_mode, search_time_range, and runtime.

    For supported predicate values, see Predicate type values in the next section on this page.

    In complex predicates, AND, OR, and NOT operators must be upper case. Lower case is not supported.

    Schedule (Optional) Set a schedule for the workload rule. The schedule determines the time period during which the rule is valid.


    If set to Always On (the default), the rule remains valid indefinitely and does not expire.

    If set to Time Range, the rule is valid during the specified time range only and expires when the time range ends.

    If set to Every Day, Every Week, or Every Month, the rule becomes valid on a recurring basis during the specified time range every day, on the specified days of the week, or on the specified days of the month.

    The schedule time for a workload rule is based on the system timezone, regardless of the timezone set for an individual user in the UI.

    Action Specify the action to perform when a search meets the predicate condition.


    Place search in a Pool (the default) assigns searches that meet the predicate condition to the specified workload pool.

    Abort search kills the search process.

    Display in Messages shows a message to users that have certain capabilities, such as list_workload_pools, in Splunk Web.

    Move search to alternate Pool moves the running search to a different specified pool.

    Abort search, Display in Messages, and Move search to alternate Pool actions apply to in-progress searches only. You must specify a runtime condition to enable these actions. For example, the predicate index=_internal AND runtime>1m triggers the specified action on all searches that contain index=_internal and run for more than one minute.

    The Place search in a pool action is not valid with rules containing a runtime condition. Abort search, Display in Messages, and Move search to alternate Pool actions are valid only when a runtime condition exists.

    Workload Pool Select the workload pool to which this rule applies.
  4. Click Submit.

Predicate type values

The following table shows valid values for each workload rule predicate type:

Predicate type Valid values
app Name of the app. For example, app=search

The correct name to specify for an app is the name of the app directory located in $SPLUNK_HOME/etc/apps. You can also find the correct name for an app in Splunk Web: Click Apps > Manage Apps. See app names listed under Folder name. App names are case insensitive.

role Name of the role. For example, role=admin.

For important details on role predicate values, see Searches run by single user can match multiple roles directly after this table on this page.

index Name of the index. For example, index=_internal. Value can refer to internal or public index. You can optionally specify index=*" to classify searches containing either index=* or index=_*.
user Name of any valid user. For example, user=bob. The reserved internal user "nobody" is invalid; the reserved internal user "splunk-system-user" is valid.
search_type adhoc, scheduled, datamodel_acceleration, report_acceleration, and summary_index
search_mode realtime and historical
search_time_range An absolute time range during which the rule is valid. Currently supports the value alltime only.
runtime The amount of time that a search must run in a workload pool to trigger a specified action, such as Abort search, Display in Messages, or Move search to alternate Pool. For example, if you specify runtime>1m, any search in the pool that runs for more than 1 minute is subject to the specified action.

By default, runtime applies to both historical and real-time searches. If you want the runtime condition to apply to historical searches only or real-time searches only, you must specify the search_mode in the predicate condition. For example, if you specify runtime>5m AND search_mode=realtime, only those realtime searches in the pool that run for more than 5 minutes are subject to the specified action.

Valid units for runtime values include s, second, seconds, m, minute, minutes, and h, hour, hours.

Searches run by a single user can match multiple roles

By default, the admin role in Splunk Enterprise inherits the user role and the power role. As a result of this role inheritance, in workload management, searches run by a single user can match to multiple roles in workload rules. For example, searches run by the "admin" user can match to workload rules that specify role=admin, role=user and role=power.

To avoid having a single user, such as the "admin" user, match to multiple roles in your workload rules, specify user=<username> in the rule predicate as an alternative to specifying role.

For more information on role inheritance, see About configuring role-based user access in Securing Splunk Enterprise.

For more information on workload rule configuration settings, see workload_rules.conf in the Admin Manual.

Create a workload rule using the CLI

To create a workload rule using the CLI, run the splunk add workload-rule command as follows, where predicate has the syntax <type>=<value> with optional AND, NOT, OR, ():

./splunk add workload-rule <rule_name> -predicate <predicate> -workload_pool <pool_name>

For example:

./splunk add workload-rule "my_role_rule" -predicate "app=search AND (NOT index=_internal)" \
-workload_pool "pool_a"

To create a workload monitoring rule, the -predicate value must include a runtime condition, and specify the -action value. For the "move" action, you must also specify the -destination value. For example:

./splunk add workload-rule "my_monitoring_rule" \
-predicate "(role!=admin AND (search_time_range=alltime OR index=*)) AND runtime>10m" \
-action <abort|move> -destination "pool_b"

It is a best practice to use parentheses to group predicate conditions for clarity.

Create a workload rule using REST

Send a POST request to the following endpoint:

workloads/rules 

For endpoint details, see workloads/rules in the REST API Reference Manual.

View workload_rules.conf

When you create a workload rule, the configuration is stored in $SPLUNK_HOME/etc/apps/<app_name>/local/workload_rules.conf.

workload_rules.conf defines the order in which rules are evaluated, the mapping of rules to workload pools, the schedule for rules, and workload monitoring rules and actions. The following is an example of a workload_rules.conf file:

[workload_rules_order]
rules = my_analyst_rule,my_app_rule,my_user_rule,my_index_rule,my_monitoring_rule

[workload_rule:my_app_rule]
predicate = app=search
workload_pool = my_app_pool

[workload_rule:my_analyst_rule]
predicate = role=analyst
workload_pool = my_analyst_pool
schedule = always_on

[workload_rule:my_user_rule]
predicate = user=admin
workload_pool = my_user_pool
schedule = always_on

[workload_rule:my_index_rule]
predicate = index=_internal
workload_pool = my_index_pool
schedule = time_range
start_time = 2019-01-01T04:00:00-08:00
end_time = 2019-01-05T04:00:00-08:00

[workload_rule:my_search_type_rule]
predicate = search_type=adhoc
workload_pool = my_adhoc_pool
schedule = every_day
start_time = 10
end_time = 15

[workload_rule:my_logical_rule_1]
predicate = app=search AND (NOT index=_internal)
workload_pool = my_logical_pool_1
schedule = every_week
start_time = 10
end_time = 23
every_week_days = [0,4,6]

[workload_rule:my_logical_rule_2]
predicate = NOT role=power OR user=admin
workload_pool = my_logical_pool_2
schedule = every_month
start_time = 1
end_time = 2
every_month_days = [1,5,16,31]

[workload_rule:my_monitoring_rule]
action = move
predicate = app=search AND runtime>1m
workload_pool = my_overflow_pool

For more information on workload rule settings, see workload_rules.conf.

Enable or disable workload management

After you create your workload pools and rules, you must enable workload management. When you initiate a request to enable workload management, a series of health checks run in the background to validate both the workload management configuration and the underlying Linux system configuration. If these health checks fail, you cannot enable workload management and a failure message appears.

For more information on Linux configuration requirements, see Set up Linux for workload management.

Enable or disable workload management in Splunk Web

  1. In Splunk Web, click Settings > Workload Management.
  2. Toggle the switch to Enabled.
    This applies any pending configuration changes and enables workload management.
    To disable workload management, toggle the switch to Disabled.

Enable or disable workload management using the CLI

To enable or disable workload management, run the following CLI command:

./splunk <enable|disable> workload-management

Enable or disable workload management using REST

You can enable or disable workload management using the REST API. For endpoint details, see workloads/config/enable or workloads/config/disable in the REST API Reference Manual.

Check workload management status

You can view the current active configuration of workload management using the CLI or REST. The output shows configuration details of all workload pools and rules, and whether workload management is supported and enabled on the instance. You can also view runtime information for in-progress searches in any pool.

Check workload management status using the CLI

Run the following CLI command:

./splunk show workload-management-status --verbose

Here is an example of the output from the command:

Workload Management Status:
	Enabled: 1
	Supported: 1
	Error:

Workload Category: ingest

		CPU Group: /sys/fs/cgroup/cpu/system.slice/Splunkd.service/ingest
		Memory Group: /sys/fs/cgroup/memory/system.slice/Splunkd.service/ingest
		CPU Weight: 25
		Memory Weight: 25
		Default Category Pool: pool_2
	pool_2:
		CPU Group: /sys/fs/cgroup/cpu/system.slice/Splunkd.service/ingest/pool_2
		Memory Group: /sys/fs/cgroup/memory/system.slice/Splunkd.service/ingest/pool_2
		CPU Weight: 100
		Memory Weight: 100

Workload Category: search

		CPU Group: /sys/fs/cgroup/cpu/system.slice/Splunkd.service/search
		Memory Group: /sys/fs/cgroup/memory/system.slice/Splunkd.service/search
		CPU Weight: 75
		Memory Weight: 75
		Default Category Pool: pool_1
	pool_1:
		CPU Group: /sys/fs/cgroup/cpu/system.slice/Splunkd.service/search/pool_1
		Memory Group: /sys/fs/cgroup/memory/system.slice/Splunkd.service/search/pool_1
		CPU Weight: 20
		Memory Weight: 20
	pool_3:
		CPU Group: /sys/fs/cgroup/cpu/system.slice/Splunkd.service/search/pool_3
		Memory Group: /sys/fs/cgroup/memory/system.slice/Splunkd.service/search/pool_3
		CPU Weight: 20
		Memory Weight: 20

Workload Category: misc

		CPU Group: /sys/fs/cgroup/cpu/system.slice/Splunkd.service/misc
		Memory Group: /sys/fs/cgroup/memory/system.slice/Splunkd.service/misc
		CPU Weight: 12
		Memory Weight: 12
		Default Category Pool: misc_pool
	misc_pool:
		CPU Group: /sys/fs/cgroup/cpu/system.slice/Splunkd.service/misc/misc_pool
		Memory Group: /sys/fs/cgroup/memory/system.slice/Splunkd.service/misc/misc_pool
		CPU Weight: 100
		Memory Weight: 100

Workload Rules:
		rule_1:
			Order: 1
			Predicate: app="search"
			Workload Pool: pool_1

		rule_2:
			Order: 2
			Predicate: app="search" AND (NOT index="_internal")
			Workload Pool: pool_3

Check workload management status using REST

To view workload management status information, send a GET request to:

workloads/status

For endpoint details, see workloads/status in the REST API Reference Manual.

PREVIOUS
Configure Linux systems not running systemd for workload management
  NEXT
Configure workload management on distributed deployments

This documentation applies to the following versions of Splunk® Enterprise: 8.0.0, 8.0.1


Was this documentation topic helpful?

Enter your email address, and someone from the documentation team will respond to you:

Please provide your comments here. Ask a question or make a suggestion.

You must be logged into splunk.com in order to post comments. Log in now.

Please try to keep this discussion focused on the content covered in this documentation topic. If you have a more general question about Splunk functionality or are experiencing a difficulty with Splunk, consider posting a question to Splunkbase Answers.

0 out of 1000 Characters