Playbook execution autoscaling

The 6.3.0 release has automatic scaling of Python runners for playbook execution.

The DECIDED daemon spawns four containerized Python runners when started. Each runner is in a container that holds a dedicated instance of the Python 3 environment.

When you upgrade Splunk SOAR (On-premises) from a lower release, the number of runners you have set is not changed.

Playbooks blocks and custom functions are assigned to different runners to increase the number of playbooks which are executed at once. This impacts how data is handled between playbook blocks and playbook runs. For more information on writing playbooks to properly share information, see Write better playbooks by following these guidelines in Build Playbooks with the Playbook Editor.

When a playbook run is started, the DECIDED daemon assigns the playbook's blocks to an available runner. Playbook blocks are assigned to an available runner as follows:

• Blocks are assigned to available runners in a round-robin fashion.
• When load increases beyond the number of available Python runners, DECIDED will spawn additional containerized Python runners.
• When load decreases, excess containers are destroyed.
• When the Runner Cycle Time is reached, the oldest existing Python runner container is destroyed, and a new container is spawned to replace it.

Playbook execution settings

Use Home Menu > Administration > Administration Settings > Playbook Execution settings to control and scale playbook execution.

• Runner Cycle Time (in minutes) - value controls the frequency with which python runners are replaced, provided there is not a conflict with other automated scaling events. This value does not control the duration of a python runner's existence, since the more python runners are active, the longer each python runner can persist before needing to be refreshed. Default is 10 minutes.
• Python Execution Block Timeout (in minutes) - this value is the maximum number of minutes that a single playbook block is allowed to run before it is terminated, and the playbook fails. Default is 60 minutes.
• Minimum number of Python 3 Runners - the minimum number of containerized Python runners. Default is four.
• Maximum number of Python 3 Runners - the maximum number of containerized Python runners. Default is 20.

When to add more Python runners

By default, starts with four containerized Python runners. Additional Python runners will be automatically created or destroyed, based on load, from a minimum number to a maximum number as set in Home Menu > Administration > Administration Settings > Playbook Execution.

Because every deployment is unique, and the factors that influence performance are varied, there are no hard rules for when, or by how much to increase the minimum or maximum number of runners for your deployment.

When deciding whether or not to add more runners, some factors that influence performance are:

• Number and kind of actions performed in your playbooks.
• Number of child playbooks or custom functions executed by playbooks.
• Actions that require responses from assets or external services.
• Available CPU resources.

If your deployment is queuing playbooks to run, and your hardware or virtual machine still has unused CPU capacity (such as idle cores, or low core usage percentages) you should consider increasing the number of playbook runners.

Increase the minimum and maximum number of runners by one and measure performance before adding additional runners. Repeat this until you either achieve the performance gains desired, reach the maximum number of runners, or encounter resource limits.

When you increase the number of Python runners you can see a decrease in the length of time it takes to complete a playbook. Many deployments can expect to see gains by adding between one and four more runners, with gains from adding additional Python runners tapering off after a total of five runners.

Not all playbooks and deployments are the same. Your results may vary based on the number of playbooks, the kinds of actions or processing each playbook is doing, the amount of CPU cores available to , and other effects.