Splunk® Enterprise

Workload Management

Splunk Enterprise version 9.0 will no longer be supported as of June 14, 2024. See the Splunk Software Support Policy for details. For information about upgrading to a supported version, see How to upgrade Splunk Enterprise.
This documentation does not apply to the most recent version of Splunk® Enterprise. For documentation on the most recent version, go to the latest release.

Configure Linux systems running systemd

If your Linux system is running systemd, you must configure systemd to manage splunkd as a service by creating a unit file that defines a cgroup hierarchy.

The following diagram illustrates the cgroup hierarchy under systemd:

The diagram shows the cgroup hierarchy on Linux machines running under systemd. 80 percent of the available system CPU and memory is reserved for splunkd.

For more information, see cgroups.

You must configure cpu and memory cgroups for workload management on all search heads and indexers.

Workload management supports Linux cgroups v1 only. If your Linux system has been upgraded to a version running cgroups v2, you must revert your system to cgroups v1 to use Workload Management in Splunk Enterprise. For additional guidance and resources, see Workload Management operation during the transition to cgroups v2.

Configure systemd to manage splunkd as a service

There are two ways to configure systemd to manage splunkd as a service:

Configuring systemd using splunk enable boot-start requires Splunk Enterprise version 7.2.2 or later.

System requirements

To enable workload management in Splunk Enterprise under systemd requires systemd version 219 or higher. For more information, see Linux operating system requirements.

Permissions requirements for systemd

systemd has the following permissions requirements:

  • Non-root users must have super user permissions to manually configure systemd on Linux.
  • Non-root users must have super user permissions to run start, stop, and restart commands under systemd.

For instructions on how to create a new user with super user permissions, see your Linux documentation. The specific steps can vary depending on the specific Linux distribution.

You must use sudo to run splunk start|stop|restart. If you do not use sudo, you must authenticate. For example:

==== AUTHENTICATING FOR org.freedesktop.systemd1.manage-units ===
Authentication is required to manage system services or units.
Multiple identities can be used for authentication:
 1.  <username_1>
 2.  <username_2>
Choose identity to authenticate as (1-2): 2
Password: 
==== AUTHENTICATION COMPLETE ===

Alternatively, you can install polkit rules to allow non-root users to run start, stop, and restart operations under systemd without using sudo, when running the enable-boot-start command to configure the systemd service automatically. For more information, see Install polkit rules to elevate user permissions.

Configure systemd manually

Follow these steps to configure systemd to manage splunkd as a service:

  1. Confirm that your Linux machine is running systemd. See Is Linux running systemd?.
  2. Before you create, delete, or modify the systemd unit file, you must stop splunkd.
    $SPLUNK_HOME/bin/splunk stop
    
  3. If you enabled Splunk software to start at boot using enable boot-start, run disable boot-start to remove the splunk init script from /etc/init.d and its symbolic links.
    sudo $SPLUNK_HOME/bin/splunk disable boot-start
    
  4. Open the $SPLUNK_HOME/etc/splunk-launch.conf file and note the value of SPLUNK_SERVER_NAME. The default value is Splunkd.
  5. In the /etc/systemd/system directory, create a unit file (text file) named <SPLUNK_SERVER_NAME>.service, such as Splunkd.service.

    You can change the SPLUNK_SERVER_NAME to any name you choose by directly editing the splunk-launch.conf file.

  6. Add the following content to the <SPLUNK_SERVER_NAME>.service unit file:
    [Unit]
    After=network.target
    
    [Service]
    Type=simple
    Restart=always
    ExecStart=/opt/splunk/bin/splunk _internal_launch_under_systemd
    KillMode=mixed
    KillSignal=SIGINT
    TimeoutStopSec=360
    LimitNOFILE=65536
    SuccessExitStatus=51 52
    RestartPreventExitStatus=51
    RestartForceExitStatus=52
    User=splunk
    Group=splunk
    Delegate=true
    CPUShares=1024
    MemoryLimit=<value>
    PermissionsStartOnly=true
    ExecStartPost=/bin/bash -c "chown -R splunk:splunk /sys/fs/cgroup/cpu/system.slice/%n"
    ExecStartPost=/bin/bash -c "chown -R splunk:splunk /sys/fs/cgroup/memory/system.slice/%n"
    
    [Install]
    WantedBy=multi-user.target
    

    The MemoryLimit value should be set to the total system memory available in bytes. The MemoryLimit value will not update if the total available system memory changes. To update the MemoryLimit value in the unit file, manually edit the unit file value and run the systemctl daemon-reload command to reload systemd.

    The following unit file properties are set specifically for Splunk workload management:
    Type=simple
    Restart=always
    Delegate=true
    Do not change these values unless you are familiar with systemd or receive guidance from Splunk support.

    Do not use the following unit file properties. These properties can cause splunkd to fail on restart.
    RemainAfterExit=yes
    ExecStop

    For more information on systemd unit file configuration settings, see Service unit configuration.

  7. Reload the unit file.
    sudo systemctl daemon-reload
    
  8. Start splunkd.
    sudo $SPLUNK_HOME/bin/splunk start
    

    This starts splunkd as a systemd service.

    Under systemd, splunk start|stop|restart commands are mapped to systemctl start|stop|restart commands.

  9. Verify that splunkd is running as a systemd service:
    sudo systemctl status <SPLUNK_SERVER_NAME>.service
    

    When you create the splunkd service, systemd creates corresponding CPU and Memory cgroups in these locations:

    CPU: /sys/fs/cgroup/cpu/system.slice/<SPLUNK_SERVER_NAME>.service
    Memory: /sys/fs/cgroup/memory/system.slice/<SPLUNK_SERVER_NAME>.service
    

    On Splunk Enterprise 8.0.0 or higher, on initial start of the splunkd service, the software checks the systemd environment, identifies the cgroup path, and automatically sets permissions for the cgroup directories.

  10. For distributed deployments, repeat steps 1-9 on all search heads and indexers.

Manage clusters under systemd

When managing an indexer cluster under systemd:

You must use the sudo command to start, stop, and restart the cluster manager node or individual peer nodes using splunk start|stop|restart commands. You do not need sudo to perform a rolling restart using the splunk rolling-restart cluster-peers command, or to take a peer offline using the splunk offline command.

When managing a search head cluster under systemd:

You must use the sudo command to start, stop, and restart cluster members using splunk start|stop|restart commands. You do not need sudo to perform a rolling restart using the splunk rolling-restart shcluster-members command, or to remove a cluster member using the splunk remove shcluster-members command.

Upgrade considerations for systemd

Upgrade from 8.0.x to 8.1

If you configured Splunk Enterprise to run as a systemd service in version 8.0.x, upon upgrade to version 8.1, Splunk Enterprise automatically updates the existing Splunkd.service unit file with the following properties:

User
Group
ExecStartPost

Upgrade from 7.3.x or earlier to 8.0

If you configured Splunk Enterprise to run as a systemd service in version 7.3.x or lower, upon upgrade to version 8.0.0, Splunk Enterprise automatically updates the existing Splunkd.service unit file. For detailed information on these unit files changes, see Upgrade considerations for systemd in the Admin Manual.

Next step

After you set up cgroups on your Linux operating system, you can configure workload management in Splunk Enterprise. See Configure workload management.

Last modified on 14 September, 2024
Set up Linux for workload management   Configure Linux systems not running systemd

This documentation applies to the following versions of Splunk® Enterprise: 8.1.0, 8.1.1, 8.1.2, 8.1.3, 8.1.4, 8.1.5, 8.1.6, 8.1.7, 8.1.8, 8.1.9, 8.1.10, 8.1.11, 8.1.12, 8.1.13, 8.1.14, 8.2.0, 8.2.1, 8.2.2, 8.2.3, 8.2.4, 8.2.5, 8.2.6, 8.2.7, 8.2.8, 8.2.9, 8.2.10, 8.2.11, 8.2.12, 9.0.0, 9.0.1, 9.0.2, 9.0.3, 9.0.4, 9.0.5, 9.0.6, 9.0.7, 9.0.8, 9.0.9, 9.0.10, 9.1.0, 9.1.1, 9.1.2, 9.1.3, 9.1.4, 9.1.5, 9.1.6, 9.2.0, 9.2.1, 9.2.2, 9.2.3


Was this topic useful?







You must be logged into splunk.com in order to post comments. Log in now.

Please try to keep this discussion focused on the content covered in this documentation topic. If you have a more general question about Splunk functionality or are experiencing a difficulty with Splunk, consider posting a question to Splunkbase Answers.

0 out of 1000 Characters