Splunk® Enterprise

Workload Management

Acrobat logo Download manual as PDF


Acrobat logo Download topic as PDF

Workload management examples

The following scenarios provide some guidance on how to use workload management in Splunk Enterprise. These are hypothetical scenarios only. The exact steps to take depend on your specific requirements.

Scenario 1: Prioritize Security team searches

Use cases:

  • Provide a high priority resource pool for all searches run by the security team.
  • Put all index=* and all time range searches in low priority pool.
  • Abort all real-time searches after 1m.
  • Move all long-running searches (>5m) that are not from the security team or admin into a low priority pool.
  • Abort all long-running searches (>10m) that are not from the security team or admin.

Steps:

  1. In Splunk Web, go to Settings > Workload Management.
  2. Click Add Workload Rule to create the following workload rule.

The order of the rules is important. Rules are evaluated in order from top to bottom. If a search triggers a rule, corresponding action is taken and none of the rules below are evaluated. For example, if Rule #2 were ordered above Rule #1 in the table below, Rule #2 will be triggered after 5 minutes and the search will be moved to alternate pool. On next evaluation, again Rule #2 will be triggered. Rule #1 will never trigger and the search will not be aborted even after 10 minutes.

Order Condition Action
1
NOT (role=security OR role=admin) AND

runtime>10m

Abort
2
NOT (role=security OR role=admin) AND

runtime>5m

Move search to alternate pool: limited_perf
3
search_mode=realtime AND

runtime>1m

Abort
4
index=* OR

search_time_range=alltime

Place search in pool:

limited_perf

5
role=security Place search in pool:

high_perf

The rules are created and placed in a certain order to achieve the use cases. The rules are evaluated every few seconds and when a new search is started. If a search meets the specified condition of a rule, the corresponding action is taken, and rules below that are not evaluated.

Scenario 2: Create a high priority pool for scheduled searches

Use cases:

  • Provide high priority pool for all scheduled searches from users in role=privileged but move these searches to the standard pool if they run for more than 2m.
  • Move all adhoc searches running for more than 5m to low priority pool.
  • Put all index=* and all time range searches in low priority pool.
  • Abort all searches running for more than 15m except searches from the admin role.

Steps:

  1. From Splunk Web, go to Settings > Workload Management.
  2. Create the following workload rules by clicking Add Workload Rule.
Order Condition Action
1 NOT (role=admin) AND

runtime>15m

Abort
2 search_type=adhoc AND

runtime>5m

Move search to alternate pool: limited_perf
3 role=privileged AND

search_type=scheduled AND runtime>2m

Move search to alternate pool: standard_perf
4 index=* OR

search_time_range=alltime

Place search in pool:

limited_perf

5 role=privileged AND

search_type=scheduled

Place search in pool:

high_perf

Scenario 3: Create admission rules to prefilter searches

Use cases:

  • Filter out a rogue search acting on all indexes or in the alltime time range.
  • Filter out a rogue search acting on all indexes and in the alltime time range and not from the Enterprise Security app.
  • Filter out an ad hoc search from a role (e.g. role=non_essential) during peak business days.
  • Filter out any search acting on the security_events index whose time range exceeds 24 hours, except for role=security_users.

Steps:

  1. In Splunk Web, click Settings > Workload Management.
  2. Click the Admission Rule tab.
  3. Create the following admission rules by clicking Add Admission Rule.
Condition Action Schedule
index=* OR search_time_range=alltime Filter search always_on
index=* AND search_time_range=alltime AND NOT app=SplunkEnterpriseSecuritySuite Filter search always_on
search_type=adhoc AND role=non_essential Filter search Every Week On

Monday, Tuesday, Wednesday, Thursday, Friday

index=security_events AND (NOT role=security_users) AND search_time_range>24h Filter search always_on

For more examples of admission rules, see Example admission rules.

Last modified on 12 October, 2023
PREVIOUS
Manually assign searches to workload pools
  NEXT
Monitor workload management using the monitoring console

This documentation applies to the following versions of Splunk® Enterprise: 8.1.0, 8.1.1, 8.1.2, 8.1.3, 8.1.4, 8.1.5, 8.1.6, 8.1.7, 8.1.8, 8.1.9, 8.1.10, 8.1.11, 8.1.12, 8.1.13, 8.1.14, 8.2.0, 8.2.1, 8.2.2, 8.2.3, 8.2.4, 8.2.5, 8.2.6, 8.2.7, 8.2.8, 8.2.9, 8.2.10, 8.2.11, 8.2.12, 9.0.0, 9.0.1, 9.0.2, 9.0.3, 9.0.4, 9.0.5, 9.0.6, 9.0.7, 9.0.8, 9.0.9, 9.1.0, 9.1.1, 9.1.2, 9.1.3, 9.2.0, 9.2.1


Was this documentation topic helpful?


You must be logged into splunk.com in order to post comments. Log in now.

Please try to keep this discussion focused on the content covered in this documentation topic. If you have a more general question about Splunk functionality or are experiencing a difficulty with Splunk, consider posting a question to Splunkbase Answers.

0 out of 1000 Characters