Splunk Cloud

Splunk Cloud User Manual

Download manual as PDF

Download topic as PDF

Workload Management

Splunk Cloud workloads perform many activities such as ingesting data, performing data model acceleration, and running searches of different types such as ad-hoc or scheduled searches concurrently. Workload management enables the allocation of resources (CPU and memory) to enable you to optimize the performance of workloads to meet your business objectives. Workload management allows you to complete the following tasks:

  • Isolate data-ingestion from the search workloads
  • Prioritize critical search workloads
  • Isolate resource-heavy searches to reduce impact to the overall system

Workload management is an administrative capability and you must have the sc_admin role to see workload pools and rules. You must have the following capabilities to configure workload managment: list_workload_pools , list_workload_rules, edit_workload_rules and select_workload_pools.


How workload management works

Workload management allocates CPU and memory resources in a logical container of resource groups called workload pools. You can then define workload rules to assign searches to different workload pools. For example, you can use workload rules to ensure that high-priority searches are placed in a workload pool that has sufficient resources, while less important searches are appropriately isolated in a different workload pool.


Workload management concepts

The following concepts and features are important to understand before you configure and use workload management.

Workload pools

A workload pool is a logical container to allow prioritization of workloads in the pool. Splunk Cloud provides three pre-defined workload pools for searches. Each pool is allocated a percentage of CPU and memory resources:

  • standard_perf: All searches are assigned to this pool by default. You must use workload rules to place searches in other pools.
  • high_perf: Compared to the standard_perf pool, this pool is assigned a larger share of system resources. Workloads assigned to this pool should result in higher performance compared to executing in the standard_perf pool.
  • limited_perf: Compared to the standard_perf pool, a relatively smaller share of system resources is assigned to this pool. Consequently, workloads assigned to this pool will execute with lower performance compared to the other two pools.



Below is the default allocation of Search resources among different pools. You cannot modify these values.

Search Category Pools (% of Search Resources):

Pool CPU Memory
standard_perf

35% 100%
high_perf

60% 100%
limited_perf

5% 100%

Workload rules

Workload rules are used to automatically assign searches to workload pools and monitor the searches after they start execution. Workload rules have a priority order in which they are evaluated. Workload rules are evaluated in order for every new search and after every few seconds.

By default, workload rules are evaluated in the order that you create them. If a search meets the conditions of a rule, the specified action is taken and no further rules are evaluated. You can edit the order of workload rules to ensure that searches matching certain predicate conditions are given priority access to specified workload pools. If a search does not meet the conditions of any rule, the search runs in the default search pool.

There are two types of workload rules:

  • Search placement rules
  • Monitoring rules and actions

Search placement rules determine whether a search is placed in the specified pool based on a user-defined predicate (condition). Predicates you can use to control search placement include app, role, user, index, search_type, search_mode, and search_time_range. You can use search placement rules to ensure that high-priority searches are assigned to pools that provide adequate resources, while low-priority searches are restricted.
Monitoring rules automatically trigger actions on running search processes based on the rule predicate and the status of the search. When you create a monitoring rule, you must specify a runtime value in the predicate. If a search exceeds the runtime value, workload management performs the specified action. Supported actions are Abort search, Display in Messages, and Move search to alternate Pool. You can use monitoring rules to manage heavy search loads and prevent rogue processes from monopolizing pool resources.

Tips

  1. When migrating to this version of Splunk Cloud, if you do nothing, there is no change in your search priority. All of your searches will run in the standard_perf pool.
  2. Selectively add workloads (by creating workload rules) to the high_perf pool to ensure higher performance and speed for that workload in your priority pool. The high_perf pool is intended to serve a few selected high priority searches. Assigning too many searches to high_perf pool will degrade the search performance.
  3. Using workload pools helps to ensure that your priority searches have high performance. This means that searches in your standard_perf and limited_perf pools may degrade somewhat by comparison. This is expected behavior, and you may need to monitor and adjust rules to ensure that you get the best performance for the searches that matter most.

Create a workload rule in Splunk Web

  1. In Splunk Web, click Settings > Workload Management.
  2. Click Add Workload Rule.
  3. Define the following fields to configure a new workload rule:
    Field Action
    Name Specify the name of the workload rule.
    Predicate (Condition) Specify a predicate (condition) that must match to trigger this rule. The predicate syntax is <type>=<value> with optional AND, OR, NOT, (). For example, app=search AND role=power triggers all searches belonging to both the Search app and the power role.

    Valid predicate types are app, role, index, user, search_type, search_mode, search_time_range, and runtime.

    For supported predicate values, see Predicate type values in the next section on this page.

    In complex predicates, AND, OR, and NOT operators must be upper case. Lower case is not supported.

    Schedule (Optional) Set a schedule for the workload rule. The schedule determines the time period during which the rule is valid.


    If set to Always On (the default), the rule remains valid indefinitely and does not expire.

    If set to Time Range, the rule is valid during the specified time range only and expires when the time range ends.

    If set to Every Day, Every Week, or Every Month, the rule becomes valid on a recurring basis during the specified time range every day, on the specified days of the week, or on the specified days of the month.

    The schedule time for a workload rule is based on the system timezone, regardless of the timezone set for an individual user in the UI.

    Action Specify the action to perform when a search meets the predicate condition. Place search in a Pool (the default) assigns searches that meet the predicate condition to the specified workload pool.


    Abort search, Display in Messages, and Move search to alternate Pool actions apply to in-progress searches only. You must specify a runtime condition to enable these actions. For example, the predicate index=_internal AND runtime>1m triggers the specified action on all searches that contain index=_internal and run for more than one minute.

    The Place search in a pool action is not valid with rules containing a runtime condition. Abort search, Display in Messages, and Move search to alternate Pool actions are valid only when a runtime condition exists.

    Workload Pool Select the workload pool to which this rule applies.
  4. Click Submit.

Predicate type values

The following table shows valid values for each workload rule predicate type:

Predicate type Valid values
app Name of the app. For example, app=search

The correct name to specify for an app is the name of the app directory located in $SPLUNK_HOME/etc/apps. You can also find the correct name for an app in Splunk Web: Click Apps > Manage Apps. See app names listed under Folder name. App names are case insensitive.

role Name of the role. For example, role=admin.

For more information on role predicate values, see Searches run by single user can match multiple roles.

index Name of the index. For example, index=_internal. Value can refer to internal or public index. You can optionally specify index=*" to classify searches containing either index= or index=_.
user Name of any valid user. For example, user=bob. The reserved internal user "nobody" is invalid; the reserved internal user "splunk-system-user" is valid.
search_type adhoc, scheduled, datamodel_acceleration, report_acceleration, and summary_index
search_mode realtime and historical
search_time_range An absolute time range during which the rule is valid. Currently supports the value alltime only.
runtime The amount of time that a search must run in a workload pool to trigger a specified action, such as Abort search, Display in Messages, or Move search to alternate Pool. For example, if you specify runtime>1m, any search in the pool that runs for more than 1 minute is subject to the specified action.

By default, runtime applies to both historical and real-time searches. If you want the runtime condition to apply to historical searches only or real-time searches only, you must specify the search_mode in the predicate condition. For example, if you specify runtime>5m AND search_mode=realtime, only those realtime searches in the pool that run for more than 5 minutes are subject to the specified action.

Valid units for runtime values include s, second, seconds, m, minute, minutes, and h, hour, hours.

Assign searches to workload pools manually

Although workload rules are a preferred way to manage search workload, you may assign searches to workload pools manually as well. Manual assignment of searches is described below.

To manually assign searches to a workload pool, you must have both the list_workload_pools and select_workload_pools capabilities.


Assign a scheduled search to a workload pool manually

You can assign a scheduled search to a workload pool using Splunk Web.

Assign a scheduled search using Splunk Web

To assign a scheduled search to a workload pool using Splunk Web, follow these steps:

  1. Click on Settings > Searches, Reports, and Alerts.
  2. Find the specific saved search, and click Edit > Advanced Edit.
  3. In the Workload Pool field, enter the name of the pool.
  4. Click Save.

Assign an ad-hoc search to a workload pool manually

You can assign an ad-hoc search to a workload pool using Splunk Web.


Assign an ad-hoc search using Splunk Web

  1. In the Search bar, enter your ad-hoc search string.
  2. Select a workload pool from the menu.
  3. Run the search.
  4. The ad-hoc search job runs in the specified workload pool. The image shows the workload pool drop-down menu on the search bar. The menu lists the available pools to which you can assign an ad-hoc search.

  5. Click Job > Inspect Job > Search job properties.
  6. Confirm that the ad-hoc search ran in the specified pool. For example:
    The image shows a list of search job properties, including the name of the workload pool in which the ad-hoc search job ran.

Assign accelerated reports to workload pools manually

You can assign any report that qualifies for acceleration to a workload pool manually.

Assigning an accelerated report to a workload pool with ample CPU and memory resources can help you minimize performance issues that can occur during report acceleration, which can be resource-intensive.

You can assign an accelerated report to a workload pool using Splunk Web.

  1. Click Settings > Searches, Reports, and Alerts.
  2. Find the report you want to accelerate and click Edit > Edit Acceleration.
  3. Select the Accelerate Report checkbox.
  4. Select the Summary Range for the report acceleration.
  5. Select a workload pool from the menu.
  6. Click Save.


For more information on report acceleration, see Accelerate reports in the Splunk Enterprise Reporting Manual.

Assign accelerated data models to workload pools manually

You can assign an accelerated data model to a workload pool using Splunk Web:

  1. Click Settings > Data models.
  2. Find the data model you want to accelerate and click Edit > Edit Acceleration.
  3. Select the Accelerate checkbox.
  4. Select the Summary Range for the data model acceleration.
  5. Select a workload pool from the menu.
  6. Click Save.

For more information on accelerated data models, see Accelerate data models in the Splunk Enterprise Knowledge Manager Manual.

Examples

These scenarios are meant to guide you on how to use workload management in Splunk Enterprise Cloud. The scenarios are just representative examples and the exact steps will depend on your specific requirements.

Scenario 1: Prioritize Security team searches

Use cases:

  • Provide a high priority resource pool for all searches run by the security team.
  • Put all index=* and all time range searches in low priority pool.
  • Abort all real-time searches after 1m.
  • Move all long-running searches (>5m) that are not from the security team or sc_admin into a low priority pool.
  • Abort all long-running searches (>10m) that are not from the security team or sc_admin.

To do this, follow the steps below:

  1. From Splunk Web, go to Settings > Workload Management.
  2. Create the following workload management rules by clicking Add Workload Rule.

The order of the rules is important. Rules are evaluated in order from top to bottom. If a search triggers a rule, corresponding action is taken and none of the rules below are evaluated. For example, if Rule #2 were ordered above Rule #1 in the table below, Rule #2 will be triggered after 5 minutes and the search will be moved to alternate pool. On next evaluation, again Rule #2 will be triggered. Rule #1 will never trigger and the search will not be aborted even after 10 minutes.

Order Condition Action
1
NOT (role=security OR role=sc_admin) AND

runtime>10m

Abort
2
NOT (role=security OR role=sc_admin) AND

runtime>5m

Move search to alternate pool: limited_perf
3
search_mode=realtime AND

runtime>1m

Abort
4
index=* OR

search_time_range=alltime

Place search in pool:

limited_perf

5
role=security Place search in pool:

high_perf

The rules are created and placed in a certain order to achieve the use cases. The rules are evaluated every few seconds and when a new search is started. If a rule is matched, the corresponding action is taken, and rules below that are not evaluated.

Scenario 2: Create a high priority pool for scheduled searches

This scenario represents the following use case:

  • Provide high priority pool for all scheduled searches from users in role=privileged but move these searches to the standard pool if they run for more than 2m.
  • Move all adhoc searches running for more than 5m to low priority pool.
  • Put all index=* and all time range searches in low priority pool.
  • Abort all searches running for more than 15m except searches from the sc_admin.

To do this, follow the steps below:

  1. From Splunk Web, go to Settings > Workload Management.
  2. Create the following workload management rules by clicking Add Workload Rule.
Order Condition Action
1 NOT (role=sc_admin) AND

runtime>15m

Abort
2 search_type=adhoc AND

runtime>5m

Move search to alternate pool: limited_perf
3 role=privileged AND

search_type=scheduled AND runtime>2m

Move search to alternate pool: standard_perf
4 index=* OR

search_time_range=alltime

Place search in pool:

limited_perf

5 role=privileged AND

search_type=scheduled

Place search in pool:

high_perf

PREVIOUS
Manage a rolling restart in Splunk Cloud
  NEXT
Upgrade your Forwarders

This documentation applies to the following versions of Splunk Cloud: 8.0.0


Was this documentation topic helpful?

Enter your email address, and someone from the documentation team will respond to you:

Please provide your comments here. Ask a question or make a suggestion.

You must be logged into splunk.com in order to post comments. Log in now.

Please try to keep this discussion focused on the content covered in this documentation topic. If you have a more general question about Splunk functionality or are experiencing a difficulty with Splunk, consider posting a question to Splunkbase Answers.

0 out of 1000 Characters