Splunk® Enterprise

Workload Management

Acrobat logo Download manual as PDF


Splunk Enterprise version 7.2 is no longer supported as of April 30, 2021. See the Splunk Software Support Policy for details. For information about upgrading to a supported version, see How to upgrade Splunk Enterprise.
Acrobat logo Download topic as PDF

Configure Linux systems not running systemd

Before you can configure workload management on Linux systems not running systemd, you must create a cgroup hierarchy in which splunkd and other system processes run in their own cgroups.

The following diagram illustrates the cgroup hierarchy on Linux systems not running systemd:

The diagram shows the cgroup hierarchy on Linux machines not running under systemd. 80 percent of the total system CPU and memory is reserved for splunkd. 20 percent of the total CPU and memory is reserved for system processes.

For more information, see cgroups.

Workload management supports Linux cgroups v1 only. If your Linux system has been upgraded to a version that runs cgroups v2 by default, you must revert your system to cgroups v1 to use workload management in Splunk Enterprise.

Configure cgroups on non-systemd distributions

There are two ways to configure cgroups for workload management on Linux systems not running systemd:

You must configure cpu and memory cgroups for workload management on all search heads and indexers.

Configure cgroups using cgconfig

You can use the cgconfig service installed with libcgroup to create cgroups on Linux systems not running systemd.

libcgroup is deprecated in RHEL 7.0 and other Linux distributions that support systemd.

To configure cgroups for workload management using cgconfig:

  1. Check that /sys/fs is mounted. If it is not mounted, mount the tmpfs in-memory filesystem under /sys/fs/cgroup as follows:
    sudo mount -t tmpfs -o size=10M tmpfs /sys/fs
    

    Splunk Enterprise examines the /proc/mounts file to determine whether your Linux machine can support workload management.

  2. Create the cgconfig.conf file under /etc and add the following contents:
    #  Copyright IBM Corporation. 2007
    #
    #  Authors: Balbir Singh <balbir@linux.vnet.ibm.com>
    #  This program is free software; you can redistribute it and/or modify it
    #  under the terms of version 2.1 of the GNU Lesser General Public License
    #  as published by the Free Software Foundation.
    #
    #  This program is distributed in the hope that it would be useful, but
    #  WITHOUT ANY WARRANTY; without even the implied warranty of
    #  MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
    #
    # See man cgconfig.conf for further details.
    #
    # By default, mount all controllers to /cgroup/<controller>
     
    mount {
        cpuset  = /sys/fs/cgroup/cpuset;
        cpu =     /sys/fs/cgroup/cpu;
        cpuacct = /sys/fs/cgroup/cpuacct;
        memory  = /sys/fs/cgroup/memory;
        devices = /sys/fs/cgroup/devices;
        freezer = /sys/fs/cgroup/freezer;
        net_cls = /sys/fs/cgroup/net_cls;
        blkio   = /sys/fs/cgroup/blkio;
    }
     
    group splunk {
        perm {
             admin {
                 uid = "splunk";
                 gid = "splunk";
             }
             task {
                 uid = "splunk";
                 gid = "splunk";
             }
         }
        cpu {
             cpu.shares = "2048";
            }
        memory {
             memory.limit_in_bytes = "3G";
               }
    }
    
  3. Restart the cgconfig service.
    # service cgconfig restart
    Stopping cgconfig service:                                 [  OK  ]
    Starting cgconfig service:                                 [  OK  ]
    

    This creates the splunk cgroup.

    The cgroup name must match the value of workload_pool_base_dir_name defined in workload_pools.conf The default value is splunk.

Configure cgroups using filesystem operations

To configure cgroups, the Linux admin, logged in as root user, must create the cgroups and assign the splunk user permissions to manage the cgroups, as follows:

  1. In the workload_pools.conf. file, set workload_pool_base_dir_name to the root cgroup to be used by splunk. For example:
    [general]
    workload_pool_base_dir_name = splunk
    

    Or, send a POST request:

    workloads/config/set-base-dirname -workload_pool_base_dir_name <base_dir_name>
    

    For endpoint details, see workloads/config/set-base-dirname in the Splunk Enterprise REST API Reference Manual.

  2. Create cpu and memory cgroups:
    sudo mkdir /sys/fs/cgroup/cpu/<workload_pool_base_dir_name>
    sudo mkdir /sys/fs/cgroup/memory/<workload_pool_base_dir_name>
    
  3. Assign the splunk user permissions to manage the respective cgroups:
    sudo chown -R ${USER} /sys/fs/cgroup/cpu/<workload_pool_base_dir_name>
    sudo chown -R ${USER} /sys/fs/cgroup/memory/<workload_pool_base_dir_name>
    

    If you have specified a user as SPLUNK_OS_USER in splunk-launch.conf, you must specify the same user as {USER} in the command. For more information see splunk-launch.conf in the Splunk Enterprise ''Admin Manual''.

  4. Assign CPU shares for splunk cgroup:
    cd /sys/fs/cgroup/cpu/splunk
    echo 2048 > cpu.shares
    
  5. Assign physical memory for the splunk cgroup:
    cd /sys/fs/cgroup/memory/splunk/
    echo 14G > memory.limit_in_bytes
    

Next step

After you set up cgroups on your Linux operating system, you can configure workload management in Splunk Enterprise. See Configure workload management.

Last modified on 23 October, 2023
PREVIOUS
Configure Linux systemd for workload management
  NEXT
Configure workload management

This documentation applies to the following versions of Splunk® Enterprise: 7.2.0, 7.2.1, 7.2.2, 7.2.3, 7.2.4, 7.2.5, 7.2.6, 7.2.7, 7.2.8, 7.2.9, 7.2.10, 7.3.0, 7.3.1, 7.3.2, 7.3.3, 7.3.4, 7.3.5, 7.3.6, 7.3.7, 7.3.8, 7.3.9, 8.0.0, 8.0.1, 8.0.2, 8.0.3, 8.0.4, 8.0.5, 8.0.6, 8.0.7, 8.0.8, 8.0.9, 8.0.10, 8.1.0, 8.1.1, 8.1.2, 8.1.3, 8.1.4, 8.1.5, 8.1.6, 8.1.7, 8.1.8, 8.1.9, 8.1.10, 8.1.11, 8.1.12, 8.1.13, 8.1.14, 8.2.0, 8.2.1, 8.2.2, 8.2.3, 8.2.4, 8.2.5, 8.2.6, 8.2.7, 8.2.8, 8.2.9, 8.2.10, 8.2.11, 8.2.12, 9.0.0, 9.0.1, 9.0.2, 9.0.3, 9.0.4, 9.0.5, 9.0.6, 9.0.7, 9.0.8, 9.1.0, 9.1.1, 9.1.2, 9.1.3, 9.2.0


Was this documentation topic helpful?


You must be logged into splunk.com in order to post comments. Log in now.

Please try to keep this discussion focused on the content covered in this documentation topic. If you have a more general question about Splunk functionality or are experiencing a difficulty with Splunk, consider posting a question to Splunkbase Answers.

0 out of 1000 Characters