Run Splunk Enterprise as a systemd service
Splunk Enterprise 7.2.2 and higher provides support for systemd
on Linux with an enhanced enable boot-start
command that lets you automatically configure systemd
to manage splunkd
as a service.
What is systemd?
systemd
is a system startup and service manager that is widely deployed as the default init system on most major Linux distributions. You can configure systemd
to manage processes, such as splunkd
, as services, and allocate system resources to those processes under cgroups
.
systemd advantages
systemd
offers the following general advantages:
- Enhanced parallel processing.
- Simplified configuration with standardized unit text files. No scripts required.
- Improved mechanism for expressing dependencies. For example, you can specify in the unit file that the network must be up before startup of the
splunkd
service occurs.
systemd
offers these additional specific advantages for Splunk deployments:
- Start
splunkd
at boot. - Monitor and manage
splunkd
service during runtime. - Provides tools to debug and troubleshoot boot-time and service activities.
- Allows more control over plug-in monitoring tools that track the status of Splunk instances.
- Simplifies the set up of
cgroups
required for workload management in Splunk Enterprise. See Set up Linux for workload management in the Workload Management manual.
Configure systemd to manage splunkd
You can use either of the following two methods to configure systemd
to manage splunkd
as a service:
If you configure systemd
using enable boot-start
, a Splunk service unit file is created automatically. No additional manual configuration is required.
System requirements
- To run
splunkd
as asystemd
service requires one of the following supported Linux distributions:- RHEL 7, 8, and 9
- CentOS 7 and 8
- Ubuntu 16.04 LTS and later
- Suse 12
- To configure
systemd
usingenable boot-start
requires Splunk Enterprise version 7.2.2 or later. - To enable workload management in Splunk Enterprise under
systemd
requiressystemd
version 219 or higher. For more information, see Linux operating system requirements in the Workload Management manual.
Workload management supports Linux cgroups v1 only. If your Linux system has been upgraded to a version running cgroups v2, you must revert your system to cgroups v1 to use Workload Management in Splunk Enterprise. For additional guidance and resources, see Workload Management operation during the transition to cgroups v2.
Permissions requirements
The enable boot-start
command and systemd
have the following permissions requirements:
- Non-root users must have super user permissions to configure
systemd
usingenable boot-start
. - Non-root users must have super user permissions to run
splunk start|stop|restart
operations undersystemd
.
For instructions on how to create a new user with super user permissions, see your Linux documentation.
Unprivileged users must use sudo
to run splunk start|stop|restart
. If you do not use sudo
and attempt to run splunk start|stop|restart
when managed by systemd, a prompt appears requesting authentication. For example:
==== AUTHENTICATING FOR org.freedesktop.systemd1.manage-units === Authentication is required to manage system services or units. Multiple identities can be used for authentication: 1. <username_1> 2. <username_2> Choose identity to authenticate as (1-2): 2 Password: ==== AUTHENTICATION COMPLETE ===
Alternately, you can install polkit rules with the enable boot-start
command to allow unprivileged users to run start|stop|restart
operations under systemd
without using sudo
. For instructions, see Install polkit rules to elevate user permissions.
Unit file naming considerations
The enable boot-start
command creates a systemd
unit file named Splunkd.service
. The unit file name is based on the SPLUNK_SERVER_NAME
in splunk-launch.conf
, which is set by default to Splunkd
.
If for any reason you remove the SPLUNK_SERVER_NAME
value from splunk-launch.conf
, enable boot-start
creates a unit file named splunkd.service
(lower case "splunkd") and sets SPLUNK_SERVER_NAME=splunkd
in the splunk-launch.conf
file.
You can specify a different name of your choice for the unit file when you run enable boot-start
. See Specify a different unit file name.
Configure systemd using enable boot-start
You can configure systemd
to manage splunkd
as a service using the enable boot-start
command, as follows:
- Log into the machine on which you want to configure
systemd
to managesplunkd
as a service. - Stop
splunkd
.$SPLUNK_HOME/bin/splunk stop
- If you previously enabled Splunk Enterprise to start at boot using the
enable boot-start
command, rundisable boot-start
to remove thesplunk
init
script located in/etc/init.d
and its symbolic links.[sudo] $SPLUNK_HOME/bin/splunk disable boot-start
For instructions on how to reinstall the
splunk
init
script, see Install splunk init script. - Run the
enable boot-start
command, specifying the-systemd-managed
,-user
, and-group
parameters, as follows:[sudo] $SPLUNK_HOME/bin/splunk enable boot-start -systemd-managed 1 -user <username> -group <groupname>
Specifying
-user
and-group
is optional but recommended. If you do not specify-user
, theSPLUNK_OS_USER
insplunk-launch.conf
is used. IfSPLUNK_OS_USER
is not defined, the owner of thesplunk
binary is used.This installs the following
systemd
service unit file, namedSplunkd.service
by default, in/etc/systemd/system
. To specify a different unit file name, use the-systemd-unit-file-name
option. See Specify a different unit file name.#This unit file replaces the traditional start-up script for systemd #configurations, and is used when enabling boot-start for Splunk on #systemd-based Linux distributions. [Unit] Description=Systemd service file for Splunk, generated by 'splunk enable boot-start' After=network.target [Service] Type=simple Restart=always ExecStart=/opt/splunk/bin/splunk _internal_launch_under_systemd KillMode=mixed KillSignal=SIGINT TimeoutStopSec=360 LimitNOFILE=65536 SuccessExitStatus=51 52 RestartPreventExitStatus=51 RestartForceExitStatus=52 User=splunk Group=splunk Delegate=true CPUShares=1024 MemoryLimit=<value> PermissionsStartOnly=true ExecStartPost=/bin/bash -c "chown -R splunk:splunk /sys/fs/cgroup/cpu/system.slice/%n" ExecStartPost=/bin/bash -c "chown -R splunk:splunk /sys/fs/cgroup/memory/system.slice/%n" [Install] WantedBy=multi-user.target
The
MemoryLimit
value is set to the total system memory available in bytes when the service unit file is created. TheMemoryLimit
value will not update if the total available system memory changes. To update theMemoryLimit
value in the unit file, you can manually edit the value or use theboot-start
command to disable and re-enable systemd.The following unit file properties are required. Do not change these values without appropriate guidance.
Type=simple
Restart=always
ExecStart=$SPLUNK_HOME/bin/splunk _internal_launch_under_systemd
Delegate=true
This property is required for workload management. See Configure workload management.Do not use the following properties. These properties can cause
splunkd
to fail on restart.
RemainAfterExit=yes
ExecStop
For more information, see Systemd unit file properties.
- Start
splunkd
.[sudo] $SPLUNK_HOME/bin/splunk start
This starts
splunkd
as asystemd
service.Under
systemd
,splunk start|stop|restart
commands are mapped tosystemctl start|stop|restart
commands. - Verify that
splunkd
is running as asystemd
service. For example:$SPLUNK_HOME/bin/splunk status splunkd is running (PID: 24772). splunk helpers are running (PIDs: 24843 24857 24984 25032).
Alternatively, you can use
systemctl status
to check if thesplunkd
process is running. However, when using this command, a brief time lag can occur during whichsystemctl status
shows "active" andsplunk status
shows "splunkd is not running".Configuring
systemd
to managesplunkd
as a service creates CPU and Memorycgroups
in these locations:CPU: /sys/fs/cgroup/cpu/system.slice/Splunkd.service Memory: /sys/fs/cgroup/memory/system.slice/Splunkd.service
- For distributed deployments, repeat steps 1-8 on all search heads and indexers.
Additional options for enable boot-start
The enable boot-start
command supports these additional options:
Install splunk init script
In version 7.2.2 and higher, the enable boot-start
command adds a -systemd-managed 0|1
option that controls whether to install the splunk
init
script in /etc/init.d
or the Splunkd.service
unit file in /etc/systemd/system
.
To install the splunk
init
script, specify -systemd-managed 0
:
$SPLUNK_HOME/bin/splunk enable boot-start -systemd-managed 0 -user <username> Init script installed at /etc/init.d/splunk. Init script is configured to run at boot.
See Configure Splunk Enterprise to start at boot time.
In version 7.2.2 through 7.2.x, if you do not specify the -systemd-managed
option, the enable boot-start
command defaults to -systemd-managed 1
and installs the Splunkd.service
unit file. In version 7.3.0 and later, this default behavior is reversed, and the enable boot-start
command defaults to -systemd-managed 0
and installs the splunk
init
file.
The init.d
boot-start script is not compatible with RHEL 8 and higher.
Specify a different unit file name
The default splunkd
unit file name is Splunkd.service
. You can specify a different name for the unit file and update the SPLUNK_SERVER_NAME
value in splunk-launch.conf
using the -systemd-unit-file-name
option. For example, to create a unit file with the name "splunk.service":
$SPLUNK_HOME/bin/splunk enable boot-start -systemd-managed 1 -systemd-unit-file-name splunk Systemd unit file installed at /etc/systemd/system/splunk.service. Configured as systemd managed service.
For more information, see Unit file naming considerations.
Install polkit rules to elevate user permissions
In version 8.1.1 and higher, the enable boot-start
command adds an option to install polkit rules that allow non-root users to run start
, stop
, and restart
operations under systemd
without using sudo
. Installing the polkit rules can reduce overhead for admins that must otherwise add unprivileged users to the sudoers
file to run these operations under systemd
.
To install polkit rules:
Run the enable boot-start
command, specifying the -create-polkit-rules
option, as follows:
./splunk enable boot-start -systemd-managed 1 -create-polkit-rules 1 -user <username>
If you previously ran enable boot-start
and specified a different user, you must change the owner of $SPLUNK_HOME
to the new user for whom you create the polkit rules. For example:
chown -R <username> $SPLUNK_HOME
Before you can install polkit rules using the create-polkit-rules
option, you must install the Polkit library on your system if you have not already done so.
Configure systemd on a clean install
To configure systemd
on a clean installation of Splunk Enterprise:
-
Expand the install package in an appropriate directory. For example:
tar xvfz splunk_package_name.tgz -C /opt
- Run
enable boot-start
to install theSplunkd.service
unit file:sudo $SPLUNK_HOME/bin/splunk enable boot-start -systemd-managed 1 -user <username>
When running enable boot-start for the first time after a clean install, Splunk Enterprise prompts you to accept the Splunk software license agreement. To automatically accept the license without prompt, specify the
--accept-license
flag with the command. -
Start
splunkd
.sudo $SPLUNK_HOME/bin/splunk start
-
Verify that
splunkd
is running as asystemd
service.$SPLUNK_HOME/bin/splunk status
Manage clusters under systemd
When managing an indexer cluster under systemd
:
- You must use the
sudo
command to start, stop, and restart the cluster manager node or individual peer nodes usingsplunk start|stop|restart
commands. - You do not need
sudo
to perform a rolling restart using thesplunk rolling-restart cluster-peers
command, or to take a peer offline using thesplunk offline
command.
When managing a search head cluster under systemd
:
- You must use the
sudo
command to start, stop, and restart cluster members usingsplunk start|stop|restart
commands. - You do not need
sudo
to perform a rolling restart using thesplunk rolling-restart shcluster-members
command, or to remove a cluster member using thesplunk remove shcluster-members
command.
Upgrade considerations for systemd
Upgrade from 8.0.x to 8.1
If you configured Splunk Enterprise version 8.0.x to run as a systemd
service, upon upgrade to version 8.1, Splunk Enterprise adds the following properties to the Splunkd.service
unit file:
User Group ExecStartPost
When Splunk Enterprise adds these unit file properties, it creates a new unit file that replaces the existing unit file Splunkd.service
. It also renames the old unit file Splunkd.service_<timestamp>
, which it saves for backup purposes only.
When upgrading directly from 7.3.x or lower to 8.1, Splunk Enterprise adds the Group
property to the unit file.
Upgrade from 7.3.x or lower to 8.0
If you configured Splunk Enterprise version 7.3.x or lower to run as a systemd
service, upon upgrade to version 8.0.x, on initial start, Splunk Enterprise modifies the existing systemd
configuration as follows:
- It removes the
ExecStartPost
andUser
properties from theSplunkd.service
unit file. - It checks the
systemd
environment, identifies thecgroup
path, and automatically sets permissions for the correctcgroup
directories.
You must use sudo splunk start
to perform the initial start of Splunk Enterprise after installing the version 8.0.0 upgrade tarball.
Using systemctl start
to perform the initial start of Splunk Enterprise on upgrade to version 8.0.0 will fail.
For detailed information on upgrading Splunk Enterprise, see How to upgrade Splunk Enterprise in the Installation Manual.
Configure Splunk Enterprise to start at boot time | Install your license |
This documentation applies to the following versions of Splunk® Enterprise: 8.1.1, 8.1.2, 8.1.3, 8.1.4, 8.1.5, 8.1.6, 8.1.7, 8.1.8, 8.1.9, 8.1.10, 8.1.11, 8.1.12, 8.1.13, 8.1.14, 8.2.0, 8.2.1, 8.2.2, 8.2.3, 8.2.4, 8.2.5, 8.2.6, 8.2.7, 8.2.8, 8.2.9, 8.2.10, 8.2.11, 8.2.12, 9.0.0, 9.0.1, 9.0.2, 9.0.3, 9.0.4, 9.0.5, 9.0.6, 9.0.7, 9.0.8, 9.0.9, 9.0.10, 9.1.0, 9.1.1, 9.1.2, 9.1.3, 9.1.4, 9.1.5, 9.1.6, 9.2.0, 9.2.1, 9.2.2, 9.2.3
Feedback submitted, thanks!