Configure systemd distributions for workload management
Before you can configure workload management on Linux distributions running
systemd, you must configure
systemd to manage
splunkd as a service by creating a unit file that defines a cgroup hierarchy.
The following diagram illustrates the cgroup hierarchy under
Linux cgroups must be properly configured for workload management on all search heads and indexers.
You must have root access to set up
Configure cgroups on systemd
Follow these steps to configure cgroups for workload management on
- Confirm that your Linux machine is running
systemd. See Is Linux running systemd?.
- If you have previously enabled Splunk software to start at boot using the
enable boot-startcommand, run
disable boot-startto remove both the
splunkstart script located in
/etc/init.dand symbolic links.
[sudo] $SPLUNK_HOME/bin/splunk disable boot-start
$SPLUNK_HOME/etc/splunk-launch.confand get the value of
SPLUNK_SERVER_NAME. The default value is
- In the
/etc/systemd/systemdirectory , create a unit file named
SPLUNK_SERVER_NAME.service. For example,
You must use the same
SPLUNK_SERVER_NAMEon all search heads and indexers.
- Add the following content to the
[Unit] Description=Splunk service Wants=network.target After=network.target [Service] Restart=always Type=simple ExecStart=$SPLUNK_HOME/bin/splunk _internal_launch_under_systemd --accept-license --no-prompt --answer-yes Delegate=true #Splunk defines successful exit codes other than 0 to indicate special exit scenarios which are #used by splunk operations like rolling-restart, offline etc. SuccessExitStatus=51 52 RestartPreventExitStatus=51 RestartForceExitStatus=52 RemainAfterExit=no #On some systemd installations, systemd does not create cgroups for memory and cpu controller under system.slice #rather it runs process under root cgroups, we can force systemd to create cgroups under system.slice by specifying #MemoryLimit and CPUShares, please look at description below. MemoryLimit=100G CPUShares=8192 #If you want to run splunk as root user, comment out the following five lines: PermissionsStartOnly=true User=splunk Group=splunk ExecStartPost=/bin/bash -c "chown -R <USER Specified above>:<GROUP of User> /sys/fs/cgroup/cpu/system.slice/%n" ExecStartPost=/bin/bash -c "chown -R <USER Specified above>:<GROUP of User> /sys/fs/cgroup/memory/system.slice/%n" [Install] WantedBy=multi-user.target
The following unit file property values are set specifically for Splunk workload management:
Delegate=true. Do not change these values unless you know what you are doing, or under guidance from Splunk support.
Do not set
RemainAfterExit=yesin the unit file. Setting this property to
yescauses failure of both single-instance restart using Splunk Web and rolling restart in search head cluster and indexer cluster deployments.
Do not use the
KillModeproperties in the unit file. These properties can also cause Splunk restart to fail.
CPUSharesto a higher value gives
splunkdmore CPU relative to other system processes.
For more information on unit file configuration settings, see Systemd unit file properties.
- Reload the unit file.
splunkddetects that it is running under systemd and automatically translates
splunk start|stopcommands to
- Verify that
splunkdis running as a
systemctl status <SPLUNK_SERVER_NAME>.service
Note that you do not need to create your own cgroups. When you create a Splunk service,
systemdcreates corresponding CPU and Memory cgroups in these locations:
CPU: /sys/fs/cgroup/cpu/system.slice/<SPLUNK_SERVER_NAME>.service Memory: /sys/fs/cgroup/memory/system.slice/<SPLUNK_SERVER_NAME>.service
- For distributed deployments, repeat steps 1-9 above on all search heads and indexers.
systemd unit file properties
The following table describes unit file properties that you must specify to run
splunkdas a service under
Property Expected Value Description Restart always Restarts the splunk service regardless of the reason the service stopped. Type simple The process configured with
ExecStart(shown below) is the main process of the service.
Commands with their arguments that are executed when the service is started. ExecStartPost chown -R <USER>:<GROUP of User>/sys/fs/cgroup/<cpu or memory>/system.slice/%n" Additional commands that are executed after the command in ExecStart. Delegate True Enables delegation of further resource control partitioning to processes of the unit. When enabled the unit can create and manage its own private sub-hierarchy of control groups below the control group of the unit itself. This gives the Splunk service permission to create and manage workload pools underneath the main service level cgroup. SuccessExitStatus 51 52 Takes a list of exit status definitions which, when returned by the main service process, will be considered successful termination, in addition to the normal successful exit code 0 and the signals SIGHUP, SIGINT, SIGTERM, and SIGPIPE. RestartPreventExitStatus 51 Takes a list of exit status definitions which, when returned by the main service process, will prevent automatic service restarts, regardless of the restart setting configured with Restart=. For the Splunk service, this value must be 51. RestartForceExitStatus 52 Takes a list of exit status definitions which, when returned by the main service process, will force automatic service restarts, regardless of the restart setting configured with Restart=. For the Splunk service, this value must be 52. This value is used for splunk commands such as
RemainAfterExit no (default) Set value to
noto ensure that
systemdnotices that processes belonging to the Splunk service have exited and the service is not active.
systemdwill then proceed to restart the Splunk service. Otherwise Splunk restart might not work as expected.
MemoryLimit Example: "12G" Specifies the limit on maximum memory usage for the executed processes. The limit determines how much process and kernel memory tasks in this unit can use. By default, memory size is specified in bytes. If the value has has a suffix of K, M, G or T, the specified memory size is parsed as Kilobytes, Megabytes, Gigabytes, or Terabytes (with the base 1024), respectively. Alternatively, you can specify a relative percentage of the installed physical memory on the system. If assigned the special value "infinity", no memory limit is applied. This controls the "memory.limit_in_bytes" control group attribute. CPUShares Example: "8192" Assigns the specified CPU time share weight to the processes executed. These options take an integer value and control the "cpu.shares" control group attribute. The allowed range is 2 to 262144. Default is 1024. User, Group <Splunk Owner> <Splunk Group> Sets the Linux user or group that the processes are executed as, respectively. Takes a single user or group name, or a numeric ID as argument. Remove the User and Group properties from the unit file to run as root.
For more information on
systemdunit file properties, see Service unit configuration.
After you set up cgroups on your Linux operating system, you can configure workload management in Splunk Enterprise. For more information, see Configure workload management.
Set up Linux for workload management
Configure Linux systems not running systemd
This documentation applies to the following versions of Splunk® Enterprise: 7.2.0, 7.2.1
Feedback submitted, thanks!