Upgrade workload management

Workload management support for Linux cgroups v2 is now available for Early Access customers in Splunk Enterprise version 9.3.0. In the Early Access release stage, Splunk products may have limitations on customer access, features, maturity and regional availability. For additional information on Early Access please contact your Splunk representative.

Workload management in Splunk Enterprise 7.3.x and higher includes an additional level in the cgroup hierarchy that facilitates separation of resources into "workload categories" based on process type. There are three workload categories: search, ingest, and misc.

When you migrate from 7.2.x to 7.3.x or higher, workload management automatically creates workload categories and places your existing workload pools under the appropriate category as follows:

• search: All existing search pools are placed under the search category. The existing default_ pool becomes the default_category_pool in the search category.
  
• ingest: The existing ingest_pool becomes the default_category_pool in the ingest category.
• misc: The misc category is created with no pools defined and no resources allocated.

For more information, see Configure workload categories.

On upgrade from 7.2.x to 7.3.x or higher, workload management implements these additional changes:

• The existing workload_pools.conf configuration file is converted to a new format that includes workload_category definitions. See workload_pools.conf.
• CPU and memory resources are recalculated for workload pools based on the amount of resources allocated to the parent workload category. CPU weights are recalculated as a ratio of the parent category's value. Memory is recalculated as a percentage of the parent category's value.

On upgrade from 7.2.x or 7.3.x to 8.0.0, the default memory resource allocation for the ingest category changes from 20% to 100%. The default memory resource allocation for the search and misc. categories remains the same at 70% and 10%, respectively.

Workload management continues to function without interruption during the migration process.

Before you upgrade from 7.2.x to 7.3.x or higher, make sure workload management is enabled in 7.2.x. If it is not enabled, make sure you have a valid 7.2.x workload_pools.conf configuration. If the 7.2.x configuration is invalid, migration does not occur. See workload_pools.conf.

Upgrade workload management on a non-clustered deployment

To upgrade workload management on a single instance or a non-clustered distributed deployment:

Upgrade each instance separately following the normal procedure for any Splunk Enterprise upgrade, as described in How to upgrade Splunk Enterprise in the Installation Manual.

During upgrade, workload management migrates to the updated feature architecture in 7.3.x and higher, and converts the existing workload_pools.conf configuration to the new file format.

Upgrade workload management on a search head cluster

To upgrade workload management on a search head cluster:

Upgrade the search head cluster using rolling upgrade. During upgrade, workload management migrates to the new format in 7.3.x and writes the changes to disk on each cluster member. See Perform a rolling upgrade of a search head cluster in the Distributed Search manual.

During rolling upgrade, do not make any edits to the workload management configuration. This can introduce errors into the updated workload management configuration due to ongoing configuration replication on the cluster .

During rolling upgrade, due to ongoing configuration replication on the cluster, the migrated 7.2.x to 7.3.x or higher configuration propagates the newly converted cpu_weight and mem_weight values back to members running the earlier version. While some deprecated stanzas from the 7.2.x format are retained in the 7.3.x and higher format to maintain backwards compatibility, these recalculated cpu_weight and memory_weight values can affect ongoing searches.

Deployer considerations for workload management migration

During rolling upgrade, as part of migration, workload management reads the existing configuration in the app's default directory and writes the updated configuration to the app's local directory on the members. After the migration, if you use the deployer to push new workload management configurations to members using the default push mode, merge_to_default, the new configurations that you push to the app's default directory will be overridden by the migrated configuration in the app's local directory due to configuration file precedence.

To avoid this behavior, push the new workload management configuration to members as follows:

1. Copy the upgraded workload_pools.conf file from the app's local directory on any member to the app's local directory on the deployer.
2. Delete the local workload_pools.conf file on each cluster member.
3. Set the deployer bundle push mode that fits your use case. See Choose a deployer push mode.
4. Push the configuration bundle from the deployer to the cluster. See Push the configuration bundle.

After migration to 7.3.x or higher, do not push workload management 7.2.x configurations to the members. Pushing 7.2.x configurations can cause workload management to fail. You must push 7.3.x or higher configurations only.

Upgrade workload management on an indexer cluster

To upgrade workload management on an indexer cluster:

1. Upgrade the cluster master, following the normal procedure for any Splunk Enterprise upgrade. See How to upgrade Splunk Enterprise in the Installation Manual.
2. Upgrade the search heads connected to the indexer cluster. If you are upgrading a search head cluster, it is best to perform a rolling upgrade. See Perform a rolling upgrade of a search head cluster in the Distributed Search manual.
3. Upgrade the peer nodes. See Upgrade an indexer cluster in Managing indexers and clusters of indexers.
   

   When you upgrade the peer nodes, workload management converts the existing configuration to the new format in 7.3.x and higher, but does not write the configuration to disk. Instead, a message appears in Splunk Web on the search heads alerting you that indexers are out of sync between memory and disk, and that a bundle push is required to update the peer node configuration, as described in steps 4 and 5.

4. Copy the upgraded configuration from any search head to the cluster master.
5. Push the configuration bundle to the peer nodes. See Distribute the configuration bundle.

Search head and indexer version compatibility

You can run workload management with search heads and cluster master on version 7.3.x or 8.0.x and indexers on 7.2.x. However, due to the updated workload_pools.conf file format as of 7.3.x, indexers running 7.2.x cannot understand the pool information sent from search heads running 7.3.x or 8.0.x. In this case, indexers run all searches in the default_pool specified in the 7.2.x version of workload_pools.conf.

For more information on version compatibility between indexer cluster components, see Splunk Enterprise version compatibility in Managing Indexers and Clusters of Indexers.