Configure Linux systems running systemd
If your Linux system is running systemd
, you must configure systemd
to manage splunkd
as a service by creating a unit file that defines a cgroup
hierarchy.
The following diagram illustrates the cgroup
hierarchy under systemd
:
For more information, see cgroups.
You must configure cpu and memory cgroups for workload management on all search heads and indexers.
Workload management supports Linux cgroups v1 only. If your Linux system has been upgraded to a version running cgroups v2, you must revert your system to cgroups v1 to use Workload Management in Splunk Enterprise. For additional guidance and resources, see Workload Management operation during the transition to cgroups v2.
Configure systemd to manage splunkd as a service
There are two ways to configure systemd
to manage splunkd
as a service:
- Configure
systemd
manually. - Configure
systemd
automatically using thesplunk enable boot-start
command.
Configuring systemd
using splunk enable boot-start
requires Splunk Enterprise version 7.2.2 or later.
System requirements
To enable workload management in Splunk Enterprise under systemd
requires systemd
version 219 or higher. For more information, see Linux operating system requirements.
Permissions requirements for systemd
systemd
has the following permissions requirements:
- Non-root users must have super user permissions to manually configure
systemd
on Linux. - Non-root users must have super user permissions to run
start
,stop
, andrestart
commands undersystemd
.
For instructions on how to create a new user with super user permissions, see your Linux documentation. The specific steps can vary depending on the specific Linux distribution.
You must use sudo
to run splunk start|stop|restart
. If you do not use sudo
, you must authenticate. For example:
==== AUTHENTICATING FOR org.freedesktop.systemd1.manage-units === Authentication is required to manage system services or units. Multiple identities can be used for authentication: 1. <username_1> 2. <username_2> Choose identity to authenticate as (1-2): 2 Password: ==== AUTHENTICATION COMPLETE ===
Alternatively, you can install polkit rules to allow non-root users to run start, stop, and restart operations under systemd without using sudo, when running the enable-boot-start
command to configure the systemd service automatically. For more information, see Install polkit rules to elevate user permissions.
Configure systemd manually
Follow these steps to configure systemd
to manage splunkd
as a service:
-
Confirm that your Linux machine is running
systemd
. See Is Linux running systemd?. -
Before you create, delete, or modify the
systemd
unit file, you must stopsplunkd
.$SPLUNK_HOME/bin/splunk stop
-
If you enabled Splunk software to start at boot using
enable boot-start
, rundisable boot-start
to remove thesplunk
init
script from/etc/init.d
and its symbolic links.sudo $SPLUNK_HOME/bin/splunk disable boot-start
-
Open the
$SPLUNK_HOME/etc/splunk-launch.conf
file and note the value ofSPLUNK_SERVER_NAME
. The default value isSplunkd
. -
In the
/etc/systemd/system
directory, create a unit file (text file) named<SPLUNK_SERVER_NAME>.service
, such asSplunkd.service
.You can change the
SPLUNK_SERVER_NAME
to any name you choose by directly editing thesplunk-launch.conf
file. -
Add the following content to the
<SPLUNK_SERVER_NAME>.service
unit file:[Unit] After=network.target [Service] Type=simple Restart=always ExecStart=/opt/splunk/bin/splunk _internal_launch_under_systemd KillMode=mixed KillSignal=SIGINT TimeoutStopSec=360 LimitNOFILE=65536 SuccessExitStatus=51 52 RestartPreventExitStatus=51 RestartForceExitStatus=52 User=splunk Group=splunk Delegate=true CPUShares=1024 MemoryLimit=<value> PermissionsStartOnly=true ExecStartPost=/bin/bash -c "chown -R splunk:splunk /sys/fs/cgroup/cpu/system.slice/%n" ExecStartPost=/bin/bash -c "chown -R splunk:splunk /sys/fs/cgroup/memory/system.slice/%n" [Install] WantedBy=multi-user.target
The
MemoryLimit
value should be set to the total system memory available in bytes. TheMemoryLimit
value will not update if the total available system memory changes. To update theMemoryLimit
value in the unit file, manually edit the unit file value and run thesystemctl daemon-reload
command to reload systemd.The following unit file properties are set specifically for Splunk workload management:
Type=simple
Restart=always
Delegate=true
Do not change these values unless you are familiar withsystemd
or receive guidance from Splunk support.Do not use the following unit file properties. These properties can cause
splunkd
to fail on restart.
RemainAfterExit=yes
ExecStop
For more information on
systemd
unit file configuration settings, see Service unit configuration. -
Reload the unit file.
sudo systemctl daemon-reload
-
Start
splunkd
.sudo $SPLUNK_HOME/bin/splunk start
This starts
splunkd
as asystemd
service.Under
systemd
,splunk start|stop|restart
commands are mapped tosystemctl start|stop|restart
commands. -
Verify that
splunkd
is running as asystemd
service:sudo systemctl status <SPLUNK_SERVER_NAME>.service
When you create the
splunkd
service,systemd
creates corresponding CPU and Memory cgroups in these locations:CPU: /sys/fs/cgroup/cpu/system.slice/<SPLUNK_SERVER_NAME>.service Memory: /sys/fs/cgroup/memory/system.slice/<SPLUNK_SERVER_NAME>.service
On Splunk Enterprise 8.0.0 or higher, on initial start of the
splunkd
service, the software checks thesystemd
environment, identifies thecgroup
path, and automatically sets permissions for thecgroup
directories. - For distributed deployments, repeat steps 1-9 on all search heads and indexers.
Manage clusters under systemd
When managing an indexer cluster under systemd
:
You must use the sudo
command to start, stop, and restart the cluster manager node or individual peer nodes using splunk start|stop|restart
commands. You do not need sudo
to perform a rolling restart using the splunk rolling-restart cluster-peers
command, or to take a peer offline using the splunk offline
command.
When managing a search head cluster under systemd
:
You must use the sudo
command to start, stop, and restart cluster members using splunk start|stop|restart
commands. You do not need sudo
to perform a rolling restart using the splunk rolling-restart shcluster-members
command, or to remove a cluster member using the splunk remove shcluster-members
command.
Upgrade considerations for systemd
Upgrade from 8.0.x to 8.1
If you configured Splunk Enterprise to run as a systemd
service in version 8.0.x, upon upgrade to version 8.1, Splunk Enterprise automatically updates the existing Splunkd.service
unit file with the following properties:
User Group ExecStartPost
Upgrade from 7.3.x or earlier to 8.0
If you configured Splunk Enterprise to run as a systemd
service in version 7.3.x or lower, upon upgrade to version 8.0.0, Splunk Enterprise automatically updates the existing Splunkd.service
unit file. For detailed information on these unit files changes, see Upgrade considerations for systemd in the Admin Manual.
Next step
After you set up cgroups on your Linux operating system, you can configure workload management in Splunk Enterprise. See Configure workload management.
Set up Linux for workload management | Configure Linux systems not running systemd |
This documentation applies to the following versions of Splunk® Enterprise: 8.1.0, 8.1.1, 8.1.2, 8.1.3, 8.1.4, 8.1.5, 8.1.6, 8.1.7, 8.1.8, 8.1.9, 8.1.10, 8.1.11, 8.1.12, 8.1.13, 8.1.14, 8.2.0, 8.2.1, 8.2.2, 8.2.3, 8.2.4, 8.2.5, 8.2.6, 8.2.7, 8.2.8, 8.2.9, 8.2.10, 8.2.11, 8.2.12, 9.0.0, 9.0.1, 9.0.2, 9.0.3, 9.0.4, 9.0.5, 9.0.6, 9.0.7, 9.0.8, 9.0.9, 9.0.10, 9.1.0, 9.1.1, 9.1.2, 9.1.3, 9.1.4, 9.1.5, 9.2.0, 9.2.1, 9.2.2, 9.3.0
Feedback submitted, thanks!