Configure systemd distributions for workload management
Before you can configure workload management on Linux distributions running systemd
, you must configure systemd
to manage splunkd
as a service by creating a unit file that defines a cgroup hierarchy.
The following diagram illustrates the cgroup hierarchy under systemd
:
Linux cgroups must be properly configured for workload management on all search heads and indexers.
You must have root access to set up systemd
.
Configure cgroups on systemd
Follow these steps to configure cgroups for workload management on systemd
:
- Confirm that your Linux machine is running
systemd
. See Is Linux running systemd?. - Stop
splunkd
.$SPLUNK_HOME/bin/splunk stop
- If you have previously enabled Splunk software to start at boot using the
enable boot-start
command, rundisable boot-start
to remove both thesplunk
start script located in/etc/init.d
and symbolic links.[sudo] $SPLUNK_HOME/bin/splunk disable boot-start
- Open
$SPLUNK_HOME/etc/splunk-launch.conf
and get the value ofSPLUNK_SERVER_NAME
. The default value isSplunkd
. - In the
/etc/systemd/system
directory , create a unit file namedSPLUNK_SERVER_NAME.service
. For example,Splunkd.service
.You must use the same
SPLUNK_SERVER_NAME
on all search heads and indexers. - Add the following content to the
Splunkd.service
unit file.[Unit] Description=Splunk service Wants=network.target After=network.target [Service] Restart=always Type=simple ExecStart=$SPLUNK_HOME/bin/splunk _internal_launch_under_systemd --accept-license --no-prompt --answer-yes Delegate=true #Splunk defines successful exit codes other than 0 to indicate special exit scenarios which are #used by splunk operations like rolling-restart, offline etc. SuccessExitStatus=51 52 RestartPreventExitStatus=51 RestartForceExitStatus=52 RemainAfterExit=no #On some systemd installations, systemd does not create cgroups for memory and cpu controller under system.slice #rather it runs process under root cgroups, we can force systemd to create cgroups under system.slice by specifying #MemoryLimit and CPUShares, please look at description below. MemoryLimit=100G CPUShares=8192 #If you want to run splunk as root user, comment out the following five lines: PermissionsStartOnly=true User=splunk Group=splunk ExecStartPost=/bin/bash -c "chown -R <USER Specified above>:<GROUP of User> /sys/fs/cgroup/cpu/system.slice/%n" ExecStartPost=/bin/bash -c "chown -R <USER Specified above>:<GROUP of User> /sys/fs/cgroup/memory/system.slice/%n" [Install] WantedBy=multi-user.target
The following unit file property values are set specifically for Splunk workload management:
Restart=always
,Type=simple
,Delegate=true
. Do not change these values unless you know what you are doing, or under guidance from Splunk support.Do not set
RemainAfterExit=yes
in the unit file. Setting this property toyes
causes failure of both single-instance restart using Splunk Web and rolling restart in search head cluster and indexer cluster deployments.Do not use the
ExecStop
andKillMode
properties in the unit file. These properties can also cause Splunk restart to fail.Setting
CPUShares
to a higher value givessplunkd
more CPU relative to other system processes.For more information on unit file configuration settings, see Systemd unit file properties.
- Reload the unit file.
systemctl daemon-reload
- Start
splunkd
$SPLUNK_HOME/bin/splunk start
This starts
splunkd
as asystemd
service.splunkd
detects that it is running under systemd and automatically translatessplunk start|stop
commands tosystemctl start|stop
commands. - Verify that
splunkd
is running as asystemd
service:systemctl status <SPLUNK_SERVER_NAME>.service
Note that you do not need to create your own cgroups. When you create a Splunk service,
systemd
creates corresponding CPU and Memory cgroups in these locations:CPU: /sys/fs/cgroup/cpu/system.slice/<SPLUNK_SERVER_NAME>.service Memory: /sys/fs/cgroup/memory/system.slice/<SPLUNK_SERVER_NAME>.service
- For distributed deployments, repeat steps 1-9 above on all search heads and indexers.
systemd unit file properties
The following table describes unit file properties that you must specify to run
splunkd
as a service undersystemd
.Property Expected Value Description Restart always Restarts the splunk service regardless of the reason the service stopped. Type simple The process configured with ExecStart
(shown below) is the main process of the service.ExecStart $SPLUNK_HOME/bin/splunk _internal_launch_under_systemd
Commands with their arguments that are executed when the service is started. ExecStartPost chown -R <USER>:<GROUP of User>/sys/fs/cgroup/<cpu or memory>/system.slice/%n" Additional commands that are executed after the command in ExecStart. Delegate True Enables delegation of further resource control partitioning to processes of the unit. When enabled the unit can create and manage its own private sub-hierarchy of control groups below the control group of the unit itself. This gives the Splunk service permission to create and manage workload pools underneath the main service level cgroup. SuccessExitStatus 51 52 Takes a list of exit status definitions which, when returned by the main service process, will be considered successful termination, in addition to the normal successful exit code 0 and the signals SIGHUP, SIGINT, SIGTERM, and SIGPIPE. RestartPreventExitStatus 51 Takes a list of exit status definitions which, when returned by the main service process, will prevent automatic service restarts, regardless of the restart setting configured with Restart=. For the Splunk service, this value must be 51. RestartForceExitStatus 52 Takes a list of exit status definitions which, when returned by the main service process, will force automatic service restarts, regardless of the restart setting configured with Restart=. For the Splunk service, this value must be 52. This value is used for splunk commands such as ./splunk offline
.RemainAfterExit no (default) Set value to no
to ensure thatsystemd
notices that processes belonging to the Splunk service have exited and the service is not active.systemd
will then proceed to restart the Splunk service. Otherwise Splunk restart might not work as expected.MemoryLimit Example: "12G" Specifies the limit on maximum memory usage for the executed processes. The limit determines how much process and kernel memory tasks in this unit can use. By default, memory size is specified in bytes. If the value has has a suffix of K, M, G or T, the specified memory size is parsed as Kilobytes, Megabytes, Gigabytes, or Terabytes (with the base 1024), respectively. Alternatively, you can specify a relative percentage of the installed physical memory on the system. If assigned the special value "infinity", no memory limit is applied. This controls the "memory.limit_in_bytes" control group attribute. CPUShares Example: "8192" Assigns the specified CPU time share weight to the processes executed. These options take an integer value and control the "cpu.shares" control group attribute. The allowed range is 2 to 262144. Default is 1024. User, Group <Splunk Owner> <Splunk Group> Sets the Linux user or group that the processes are executed as, respectively. Takes a single user or group name, or a numeric ID as argument. Remove the User and Group properties from the unit file to run as root. For more information on
systemd
unit file properties, see Service unit configuration.Next step
After you set up cgroups on your Linux operating system, you can configure workload management in Splunk Enterprise. For more information, see Configure workload management.
Set up Linux for workload management | Configure Linux systems not running systemd |
This documentation applies to the following versions of Splunk® Enterprise: 7.2.0, 7.2.1
Feedback submitted, thanks!