Configure cgroups v2 in Splunk Enterprise

Splunk Enterprise 9.3 and higher supports workload management on Linux systems that use cgroups version 1 (cgroups v1) and cgroups version 2 (cgroups v2).

Splunk Enterprise 9.3.x supports cgroups v1 by default. You can use workload management in Splunk Enterprise 9.3 with cgroups v1 as is. No additional action is required.

If your Linux system uses cgroups v2 by default, you must enable cgroups v2 in Splunk Enterpirse 9.3 before you can use workload management. See Configure cgroups v2 in Splunk Enterprise 9.3.

Requirements

To configure Splunk Enterprise for Linux cgroups v2:

• You must have Splunk Enterprise version 9.3.x.
• Splunk Enterprise must be installed on a Linux OS running systemd with cgroups v2 enabled by default, such as RHEL 9.

Splunk Cloud Platform does not support cgroups v2 configuration.

Configure cgroups v2 in Splunk Enterprise 9.3

To configure cgroups v2 in Splunk Enterprise 9.3:

1. Stop Splunk

   ./spunk stop
   

2. Edit $SPLUNK_HOME/etc/system/local/server.conf.
3. Under the [general]stanza, add cgroup_location=auto.
4. Enable systemd boot start for Splunk Enterprise. For example:

   sudo ./splunk enable boot-start -systemd-managed 1
   

   For more information, see Run Splunk Enterprise a systemd service in the Splunk Enterprise Admin Manual.

5. Start splunkd.

   sudo systemctl start Splunkd
   

6. Configure workload management as usual, for example using Splunk Web. See Configure workload pools.

Configure cgroups v2 on Linux OS upgrade

To configure cgroups v2 in Splunk Enterprise upon upgrade to a Linux OS running cgroups v2 by default:

1. Stop Splunk:

   ./spunk stop
   

2. Disable the Splunk Enterprise boot start script (splunk.init):

   sudo ./splunk disable boot-start
   

3. Upgrade your Linux OS to a version running systemd and cgroups v2 by default. For details, consult your Linux OS documentation.
4. Edit $SPLUNK_HOME/etc/system/local/server.conf.
5. Under the [general] stanza, add cgroup_location=auto.
6. Enable systemd boot start for Splunk Enterprise. For example:

   sudo ./splunk enable boot-start -systemd-managed 1
   

   For more information, see Run Splunk Enterprise a systemd service in the Splunk Enterprise Admin Manual.

7. Start splunkd:

   sudo systemctl start Splunkd
   

8. Configure workload management as usual, for example using Splunk Web. For more information, see Configure workload pools.

Verify cgroups v2 configuration

To verify cgroups v2 configuration for Workload Management:

1. Start Splunk Enterprise as a systemd service:

   sudo systemctl start Splunkd
   

2. Run the workload management status command.

   ./splunk show workload-management-status --verbose
   

   Review the command output to verify workload management is enabled and supported.

          Workload Management Status:
   
                  Enabled: 1
   
                  Supported: 1
   
                  Error:
   

   Look for lines that show V2 preceding the cgroup path. For example:

   CPU Group: V2:/sys/fs/cgroup:/system.slice/Splunkd.service:/sys/fs/cgroup:/system.slice/Splunkd.service:/ingest:cpu
   Memory Group: V2:/sys/fs/cgroup:/system.slice/Splunkd.service:/sys/fs/cgroup:/system.slice/Splunkd.service:/ingest:memory
   

Rollback cgroups v2 to cgroups v1

To rollback the cgroups v2 configuration to cgroups v1 in Splunk Enterprise:

1. Stop Splunk

   ./spunk stop
   

2. Disable the Splunk Enterprise boot start script (splunk.init).

   sudo ./splunk disable boot-start
   

3. Reconfigure the Linux OS to cgroups v1. Restart if required. For more information, see your Linux OS documentation.
4. Enable systemd boot start for Splunk Enterprise. For example:

   sudo ./splunk enable boot-start -systemd-managed 1
   

   For more information, see Run Splunk Enterprise a systemd service in the Splunk Enterprise Admin Manual.

5. Start splunkd.

   sudo systemctl start Splunkd