Splunk Cloud

Splunk Cloud User Manual

Download manual as PDF

Download topic as PDF

Workload Management

Splunk Cloud workloads perform many activities such as ingesting data, performing data model acceleration, and running searches of different types such as ad hoc or scheduled searches concurrently. Workload management enables the allocation of resources (CPU and memory) to enable you to optimize the priority of workloads to meet your business objectives. Workload management allows you to complete the following tasks:

  • Isolate data-ingestion from the search workloads
  • Prioritize critical search workloads
  • Isolate resource-heavy searches to reduce impact to the overall system

Workload management is an administrative capability and you must have the sc_admin role to see workload pools and rules. You must have the following capabilities to configure workload managment: list_workload_pools , list_workload_rules, edit_workload_rules and select_workload_pools.

How workload management works

Workload management allocates CPU and memory resources in a logical container of resource groups called workload pools. You can then define workload rules to place searches in different workload pools automatically. For example, you can create a workload rule to ensure that your high-priority searches are placed in a workload pool that has sufficient resources, while less important searches are appropriately isolated in a different workload pool.

Workload management concepts

The following concepts and features are important to understand before you configure and use workload management:

Workload pools

A workload pool is a logical container that allows prioritization of workloads in the pool. Splunk Cloud provides three pre-defined workload pools for searches. Each pool is allocated a percentage of CPU and memory resources:

  • standard_perf: All searches are assigned to this pool by default. You must use workload rules to place searches in other pools.
  • high_perf: Compared to the standard_perf pool, this pool is assigned a larger share of system resources. Workloads assigned to this pool are assigned a higher priority compared to executing in the standard_perf pool when system resources are in contention. However, you might still need to modify the search for better performance. For information about search optimization, see Search Optimization in the Search manual.
  • limited_perf: Compared to the standard_perf pool, a relatively smaller share of system resources is assigned to this pool. Consequently, workloads assigned to this pool will execute with the lowest priority compared to the other two pools.

The following table shows the default allocation of Search resources among different pools. You cannot modify these values.

Search Category Pools (% of Search Resources):

Pool CPU Memory
standard_perf

35% 100%
high_perf

60% 100%
limited_perf

5% 100%

Workload rules

Workload rules are used to assign searches to workload pools automatically and monitor searches after they start execution. Workload rules have a priority order in which they are evaluated. Workload rules are evaluated in order for every new search and after every few seconds.

By default, workload rules are evaluated in the order that you create them. If a search meets the conditions of a rule, the specified action is taken and no further rules are evaluated. You can edit the order of workload rules to ensure that searches matching certain predicate conditions are given priority access to specified workload pools. If a search does not meet the conditions of any rule, the search runs in the default search pool.

There are two types of workload rules:

  • Search placement rules
  • Monitoring rules and actions

Search placement rules determine whether a search is placed in the specified pool based on a user-defined predicate (condition). Predicates you can use to control search placement include app, role, user, index, search_type, search_mode, and search_time_range. You can use search placement rules to ensure that high-priority searches are assigned to pools that provide adequate resources, while low-priority searches are restricted.

Monitoring rules automatically trigger actions on running searches based on the rule predicate and the status of the search. When you create a monitoring rule, you must specify a runtime value in the predicate. If a search exceeds the runtime value, workload management performs the specified action. Supported actions are Abort search, Display in Messages, and Move search to alternate Pool. You can use monitoring rules to manage heavy search loads and prevent rogue processes from monopolizing pool resources.

For more information on workload rules, see Create a workload rule.

Tips

  1. When migrating to this version of Splunk Cloud, if you do nothing, there is no change in your search priority. All of your searches will run in the standard_perf pool.
  2. Selectively add workloads (by creating workload rules) to the high_perf pool to ensure higher performance and speed for that workload in your priority pool. The high_perf pool is intended to serve a few selected high priority searches. Assigning too many searches to high_perf pool will degrade the search performance.
  3. Using workload pools helps to ensure that your priority searches have high performance. This means that searches in your standard_perf and limited_perf pools may degrade somewhat by comparison. This is expected behavior, and you may need to monitor and adjust rules to ensure that you get the best performance for the searches that matter most.

Admission rules

Admission rules filter out searches automatically before they start based on a user-defined predicate (condition).

You can use admission rules to prevent the execution of rogue searches that might consume a large amount of resources and interfere with critical search workloads. You can also use admission rules to limit which roles, apps, and so on, can run searches over specific time ranges, such as peak business days.

For more information on admission rules, see Create an admission rule.

Create a workload rule

To create a workload rule in Splunk Web:

  1. In Splunk Web, click Settings > Workload Management.
  2. Click Add Workload Rule.
  3. Define the following fields to configure a new workload rule:
    Field Action
    Name Specify the name of the workload rule.
    Predicate (Condition) Specify a predicate (condition) that must match to trigger this rule. The predicate syntax is <type>=<value> with optional AND, OR, NOT, (). For example, app=search AND role=power triggers all searches belonging to both the Search app and the power role.

    Valid predicate types are app, role, index, user, search_type, search_mode, search_time_range, and runtime.

    For supported predicate values, see Predicate type values in the next section on this page.

    In complex predicates, AND, OR, and NOT operators must be upper case. Lower case is not supported.

    Schedule (Optional) Set a schedule for the workload rule. The schedule determines the time period during which the rule is valid.


    If set to Always On (the default), the rule remains valid indefinitely and does not expire.

    If set to Time Range, the rule is valid during the specified time range only and expires when the time range ends.

    If set to Every Day, Every Week, or Every Month, the rule becomes valid on a recurring basis during the specified time range every day, on the specified days of the week, or on the specified days of the month.

    The schedule time for a workload rule is based on the system timezone, regardless of the timezone set for an individual user in the UI.

    Action Specify the action to perform when a search meets the predicate condition. Place search in a Pool (the default) assigns searches that meet the predicate condition to the specified workload pool.


    Abort search, Display in Messages, and Move search to alternate Pool actions apply to in-progress searches only. You must specify a runtime condition to enable these actions. For example, the predicate index=_internal AND runtime>1m triggers the specified action on all searches that contain index=_internal and run for more than one minute.

    The Place search in a pool action is not valid with rules containing a runtime condition. Abort search, Display in Messages, and Move search to alternate Pool actions are valid only when a runtime condition exists.

    Workload Pool Select the workload pool to which this rule applies.
    User Message Enter a custom message that notifies the end user when a search triggers the workload rule action. For example, "Search runtime exceeded 30 seconds. The search was moved to the high_perf pool."


    A user message is required with the Display in Messages action, and is optional for other actions. Messages are limited to a maximum of 140 characters.

    When a search triggers the rule action, the user message appears in the Jobs manager in Splunk Web: Click Activity > Jobs > Job. It also appears under the Job menu in the Search app.

  4. Click Submit.

    Predicate type values

    The following table shows valid values for each workload rule predicate type:

    Predicate type Valid values
    app Name of the app. For example, app=search

    The correct name to specify for an app is the name of the app directory located in $SPLUNK_HOME/etc/apps. You can also find the correct name for an app in Splunk Web: Click Apps > Manage Apps. See app names listed under Folder name. App names are case insensitive.

    role Name of the role. For example, role=admin.

    For more information on role predicate values, see Searches run by single user can match multiple roles.

    index Name of the index. For example, index=_internal. Value can refer to internal or public index. You can optionally specify index=* to classify searches containing either index= or index=_.
    user Name of any valid user. For example, user=bob. The reserved internal user "nobody" is invalid; the reserved internal user "splunk-system-user" is valid.
    search_type adhoc, scheduled, datamodel_acceleration, report_acceleration, and summary_index
    search_mode realtime and historical
    search_time_range An absolute time range during which the rule is valid. Currently supports the value alltime only.
    runtime The amount of time that a search must run in a workload pool to trigger a specified action, such as Abort search, Display in Messages, or Move search to alternate Pool. For example, if you specify runtime>1m, any search in the pool that runs for more than 1 minute is subject to the specified action.

    By default, runtime applies to both historical and real-time searches. If you want the runtime condition to apply to historical searches only or real-time searches only, you must specify the search_mode in the predicate condition. For example, if you specify runtime>5m AND search_mode=realtime, only those realtime searches in the pool that run for more than 5 minutes are subject to the specified action.

    Valid units for runtime values include s, second, seconds, m, minute, minutes, and h, hour, hours.

    Note: runtime applies to workload rules only. It is not a valid predicate type for admission rules.

    For workload rule use case examples, see Examples.

    Monitor triggered workload rule actions

    When a running search triggers a workload rule action, information about the action appears in the Search job inspector. This includes the action that was taken on the search and the timestamp. If a search triggers multiple rules, the information appears in reverse chronological order.

    To view details of a workload rule action:

    1. In Splunk Web, click Activity > Jobs.
    2. Find the specific search job and click Job > Inspect Job > Search job properties.
    3. View details of the workload rule action under the workload_action_information property.
      The image shows the details of a triggered workload rule action in the Search job inspector.

    To view the workload_action_information property, you must have list_workload_pools and select_workload_pools capabilities.

    Assign searches to workload pools manually

    Although workload rules are the preferred way to manage search workload, you can assign searches to workload pools manually as well. Manual assignment of searches is described below.

    To assign searches to workload pools manually, you must have list_workload_pools and select_workload_pools capabilities.

    Assign a scheduled search to a workload pool manually

    You can assign a scheduled search to a workload pool using Splunk Web, as follows:

    1. Click Settings > Searches, Reports, and Alerts.
    2. Find the specific saved search, and click Edit > Advanced Edit.
    3. In the Workload Pool field, enter the name of the pool.
    4. Click Save.

    Assign an ad hoc search to a workload pool manually

    You can assign an ad hoc search to a workload pool using Splunk Web, as follows:

    1. In the Search bar, enter your ad hoc search string.
    2. Select a workload pool from the menu.
    3. Run the search.
      The ad hoc search job runs in the specified workload pool.
      The image shows the workload pool drop-down menu on the search bar. The menu lists the available pools to which you can assign an ad hoc search.
    4. Click Job > Inspect Job > Search job properties.
    5. Confirm that the ad hoc search ran in the specified pool. For example:
      The image shows a list of search job properties, including the name of the workload pool in which the ad hoc search job ran.

    Assign accelerated reports to workload pools manually

    You can assign any report that qualifies for acceleration to a workload pool manually.

    Assigning an accelerated report to a workload pool with ample CPU and memory resources can help you minimize performance issues that can occur during report acceleration, which can be resource-intensive.

    To assign an accelerated report to a workload pool using Splunk Web:

    1. Click Settings > Searches, Reports, and Alerts.
    2. Find the report you want to accelerate and click Edit > Edit Acceleration.
    3. Select the Accelerate Report checkbox.
    4. Select the Summary Range for the report acceleration.
    5. Select a workload pool from the menu.
    6. Click Save.

    For more information on report acceleration, see Accelerate reports in the Splunk Enterprise Reporting Manual.

    Assign accelerated data models to workload pools manually

    You can assign an accelerated data model to a workload pool using Splunk Web, as follows:

    1. Click Settings > Data models.
    2. Find the data model you want to accelerate and click Edit > Edit Acceleration.
    3. Select the Accelerate checkbox.
    4. Select the Summary Range for the data model acceleration.
    5. Select a workload pool from the menu.
    6. Click Save.

    For more information on accelerated data models, see Accelerate data models in the Splunk Enterprise Knowledge Manager Manual.

    Create an admission rule

    Admission rules let you filter out searches automatically before they start, based on a predicate (condition) that you define. If a search meets the specified condition, the search is not executed. Predicates you can specify to filter out searches include app, role, user, index, search_type, search_mode, and search_time_range.

    You can use admission rules to prevent the execution of rogue searches, such as poorly written and potentially harmful searches that might consume an excessive amount of resources and interfere with critical search workloads. For example, you can create a rule to filter out wildcard searches that target all indexes, or filter out searches in the alltime time range.

    You can also use admission rules to set up time-bound access to searches for roles, users, apps and so on. For example, you can create a rule that filters out all ad hoc searches from a certain role during peak business days, but allows the same role to run searches on weekends.

    Admission rules have no explicit ordering. All admission rules are evaluated when a search is dispatched. If a search meets the conditions of a rule, the rule takes effect before the search is executed. If a search is already running, and you create a new admission rule that applies to that search, the running search is not affected by the new rule.

    Admission rules are enabled by default for the sc_admin role. You can create a maximum of 100 admission rules per Splunk cloud deployment.

    To create and edit admission rules, you must have the list_workload_rule and edit_workload_rule capabilities.

    To create an admission rule in Splunk Web:

    1. In Splunk Web, click Settings > Workload Management.
    2. Click Add Admission Rule.
    3. Define the following fields to configure a new admission rule:
      Field Action
      Name Specify the name of the admission rule.
      Predicate (Condition) Specify a predicate (condition) that must match to trigger this rule. The predicate syntax is <type>=<value> with optional AND, OR, NOT, (). For example, app=search AND role=power triggers all searches belonging to both the Search app and the power role.

      Valid predicate types are app, role, index, user, search_type, search_mode, and search_time_range.

      For supported predicate values, see Predicate type values on this page.

      In complex predicates, AND, OR, and NOT operators must be upper case. Lower case is not supported.

      Schedule (Optional) Set a schedule for the admission rule. The schedule determines the time period during which the rule is valid.


      If set to Always On (the default), the rule remains valid indefinitely and does not expire.

      If set to Time Range, the rule is valid during the specified time range only and expires when the time range ends.

      If set to Every Day, Every Week, or Every Month, the rule becomes valid on a recurring basis during the specified time range every day, on the specified days of the week, or on the specified days of the month.

      The schedule time for an admission rule is based on the system timezone, regardless of the timezone set for an individual user in the UI.

      Action Admission rules currently support the default Filter search action only.
      User Message Enter a custom message that notifies the end user when a search triggers the admission rule action. For example, "This search meets specified admission rule conditions. The search was not executed."


      If an ad hoc search triggers the rule action, the custom message appears beneath the search bar in the Search and Reporting app. If a scheduled search triggers the action, a default message appears in scheduler.log only.

    4. Click Submit.

    For admission rule use case examples, see Scenario 3: Create admission rules to prefilter searches.

    Examples

    These scenarios are meant to guide you on how to use workload management in Splunk Enterprise Cloud. The scenarios are just representative examples and the exact steps will depend on your specific requirements.

    Scenario 1: Prioritize Security team searches

    Use cases:

    • Provide a high priority resource pool for all searches run by the security team.
    • Put all index=* and all time range searches in low priority pool.
    • Abort all real-time searches after 1m.
    • Move all long-running searches (>5m) that are not from the security team or sc_admin into a low priority pool.
    • Abort all long-running searches (>10m) that are not from the security team or sc_admin.

    To do this, follow the steps below:

    1. From Splunk Web, go to Settings > Workload Management.
    2. Create the following workload rules by clicking Add Workload Rule.

    The order of the rules is important. Rules are evaluated in order from top to bottom. If a search triggers a rule, corresponding action is taken and none of the rules below are evaluated. For example, if Rule #2 were ordered above Rule #1 in the table below, Rule #2 will be triggered after 5 minutes and the search will be moved to alternate pool. On next evaluation, again Rule #2 will be triggered. Rule #1 will never trigger and the search will not be aborted even after 10 minutes.

    Order Condition Action
    1
    NOT (role=security OR role=sc_admin) AND

    runtime>10m

    Abort
    2
    NOT (role=security OR role=sc_admin) AND

    runtime>5m

    Move search to alternate pool: limited_perf
    3
    search_mode=realtime AND

    runtime>1m

    Abort
    4
    index=* OR

    search_time_range=alltime

    Place search in pool:

    limited_perf

    5
    role=security Place search in pool:

    high_perf

    The rules are created and placed in a certain order to achieve the use cases. The rules are evaluated every few seconds and when a new search is started. If a rule is matched, the corresponding action is taken, and rules below that are not evaluated.

    Scenario 2: Create a high priority pool for scheduled searches

    This scenario represents the following use case:

    • Provide high priority pool for all scheduled searches from users in role=privileged but move these searches to the standard pool if they run for more than 2m.
    • Move all adhoc searches running for more than 5m to low priority pool.
    • Put all index=* and all time range searches in low priority pool.
    • Abort all searches running for more than 15m except searches from the sc_admin.

    To do this, follow the steps below:

    1. From Splunk Web, go to Settings > Workload Management.
    2. Create the following workload rules by clicking Add Workload Rule.
    Order Condition Action
    1 NOT (role=sc_admin) AND

    runtime>15m

    Abort
    2 search_type=adhoc AND

    runtime>5m

    Move search to alternate pool: limited_perf
    3 role=privileged AND

    search_type=scheduled AND runtime>2m

    Move search to alternate pool: standard_perf
    4 index=* OR

    search_time_range=alltime

    Place search in pool:

    limited_perf

    5 role=privileged AND

    search_type=scheduled

    Place search in pool:

    high_perf

    Scenario 3: Create admission rules to prefilter searches

    Use cases:

    • Filter out a rogue search acting on all indexes or in the alltime time range.
    • Filter out a rogue search acting on all indexes and in the alltime time range and not from the Enterprise Security app.
    • Filter out an ad hoc search from a role (e.g. role=non_essential) during peak business days.

    To do this, follow the steps below:

    1. In Splunk Web, click Settings > Workload Management.
    2. Click the Admission Rule tab.
    3. Create the following admission rules by clicking Add Admission Rule.
    Condition Action Schedule
    index=* OR search_time_range=alltime Filter search always_on
    index=* AND search_time_range=alltime AND NOT app=es Filter search always_on
    search_type=adhoc AND role=non_essential Filter search Every Week On

    Monday, Tuesday, Wednesday, Thursday, Friday

Last modified on 16 June, 2020
PREVIOUS
Manage a rolling restart in Splunk Cloud
  NEXT
Upgrade your Forwarders

This documentation applies to the following versions of Splunk Cloud: 8.0.2004, 8.0.2006


Was this documentation topic helpful?

Enter your email address, and someone from the documentation team will respond to you:

Please provide your comments here. Ask a question or make a suggestion.

You must be logged into splunk.com in order to post comments. Log in now.

Please try to keep this discussion focused on the content covered in this documentation topic. If you have a more general question about Splunk functionality or are experiencing a difficulty with Splunk, consider posting a question to Splunkbase Answers.

0 out of 1000 Characters