Run Splunk Enterprise as a systemd service
Splunk Enterprise 7.2.2 and later adds broad support for systemd
on Linux with an updated enable boot-start
command that lets you automatically configure systemd
to manage splunkd
as a service.
What is systemd?
systemd
is a system startup and service manager that is widely deployed as the default init system on most major Linux distributions. You can configure systemd
to manage processes, such as splunkd
, as services, and allocate system resources to those processes under cgroups.
systemd advantages
systemd
offers the following general advantages:
- Enhanced parallel processing.
- Simplified configuration with standardized unit text files. No scripts required.
- Improved mechanism for expressing dependencies. For example, you can specify in the unit file that the network must be up before startup of the
splunkd
service occurs.
systemd
offers these additional specific advantages for Splunk deployments:
- Start
splunkd
at boot. - Monitor and manage
splunkd
service during runtime. - Provides tools to debug and troubleshoot boot-time and service activities.
- Allows more control over plug-in monitoring tools that track the status of Splunk instances.
- Simplifies the set up of cgroups required for workload management in Splunk Enterprise. See Set up Linux for workload management.
Configure systemd to manage splunkd
There are two ways to configure systemd
to manage splunkd
as a service:
System requirements
- To run
splunkd
as asystemd
service requires one of the following supported Linux distributions:- RHEL 7 and 8
- CentOS 7 and 8
- Ubuntu 16.04 LTS and later
- Suse 12
- To configure
systemd
usingenable boot-start
requires Splunk Enterprise version 7.2.2 or later. - To enable workload management in Splunk Enterprise under
systemd
requiressystemd
version 219 or higher. For more information, see Linux operating system requirements in the Workload Management manual.
Permissions requirements
The enable boot-start
command and systemd
have the following permissions requirements:
- Non-root users must have super user permissions to configure
systemd
usingenable boot-start
. - Non-root users must have super user permissions to run
start
,stop
, andrestart
commands undersystemd
.
For instructions on how to create a new user with super user permissions, see your Linux documentation. The specific steps might vary depending on the specific Linux distribution.
You must use sudo
to run systemctl start|stop|restart
commands. If you do not use sudo
, you must authenticate. For example:
==== AUTHENTICATING FOR org.freedesktop.systemd1.manage-units === Authentication is required to manage system services or units. Multiple identities can be used for authentication: 1. <username_1> 2. <username_2> Choose identity to authenticate as (1-2): 2 Password: ==== AUTHENTICATION COMPLETE ===
Unit file naming considerations
The enable boot-start
command creates a systemd
unit file named Splunkd.service
. The unit file name is based on the SPLUNK_SERVER_NAME
in splunk-launch.conf
, which is set by default to Splunkd
.
If for any reason you remove the SPLUNK_SERVER_NAME
value from splunk-launch.conf
, enable boot-start
creates a unit file named splunkd.service
(lower case "splunkd") and sets SPLUNK_SERVER_NAME=splunkd
in the splunk-launch.conf
file.
You can specify a different name for the unit file when you create the unit file with enable boot-start
. See Specify the unit file name.
Configure systemd using enable boot-start
You can configure systemd
to manage splunkd
as a service using the enable boot-start
command.
- Log into the machine on which you want to configure
systemd
to managesplunkd
as a service. - Stop
splunkd
.$SPLUNK_HOME/bin/splunk stop
- If you previously enabled Splunk Enterprise to start at boot using the
enable boot-start
command, rundisable boot-start
to remove thesplunk
init
script located in/etc/init.d
and its symbolic links.[sudo] $SPLUNK_HOME/bin/splunk disable boot-start
- Run the
enable boot-start
command to install thesplunkd
unit file .[sudo] $SPLUNK_HOME/bin/splunk enable boot-start -user <username>
This installs the following default unit file named
Splunkd.service
in/etc/systemd/system
. To specify a different unit file name, use the-systemd-unit-file-name
option. See Specify the unit file name.#This unit file replaces the traditional start-up script for systemd #configurations, and is used when enabling boot-start for Splunk on #systemd-based Linux distributions. [Unit] Description=Systemd service file for Splunk, generated by 'splunk enable boot-start' After=network.target [Service] Type=simple Restart=always ExecStart=/opt/splunk/bin/splunk _internal_launch_under_systemd LimitNOFILE=65536 SuccessExitStatus=51 52 RestartPreventExitStatus=51 RestartForceExitStatus=52 User=<username> Delegate=true MemoryLimit=<value> CPUShares=1024 PermissionsStartOnly=true ExecStartPost=/bin/bash -c "chown -R <username>:<username> /sys/fs/cgroup/cpu/system.slice/%n" ExecStartPost=/bin/bash -c "chown -R <username>:<username> /sys/fs/cgroup/memory/system.slice/%n" [Install] WantedBy=multi-user.target
Regarding these lines in the unit file:
ExecStartPost=/bin/bash -c "chown -R <username>:<username> /sys/fs/cgroup/cpu/system.slice/%n" ExecStartPost=/bin/bash -c "chown -R <username>:<username> /sys/fs/cgroup/memory/system.slice/%n"
if a group does not exist on they system with the name "username", the
splunkd
service will not start. To workaround this issue, manually update theSplunkd.service
unit file with the correct group name after runningenable boot-start
.If you run
enable boot-start
as root without specifying-user
, the default unit file appears as follows:[Unit] Description=Systemd service file for Splunk, generated by 'splunk enable boot-start' After=network.target [Service] Type=simple Restart=always ExecStart=/opt/splunk/bin/splunk _internal_launch_under_systemd LimitNOFILE=65536 SuccessExitStatus=51 52 RestartPreventExitStatus=51 RestartForceExitStatus=52 Delegate=true MemoryLimit=<value> CPUShares=1024 [Install] WantedBy=multi-user.target
The
MemoryLimit
value should be set to the total system memory available in bytes. TheMemoryLimit
value will not update if the total available system memory changes. To update theMemoryLimit
value in the unit file, manually edit the unit file value and run the systemctl daemon-reload command to reload systemd. - After creating the unit file with
enable boot-start
, to ensure graceful shutdown, add these additional properties to the[Service]
stanza of the unit file:KillMode=mixed KillSignal=SIGINT TimeoutStopSec=10min
The following unit file properties are required. Do not change these values without appropriate guidance.
Type=simple
Restart=always
ExecStart=$SPLUNK_HOME/bin/splunk _internal_launch_under_systemd
Delegate=true
This property is required for workload management. See Configure workload management.Do not use the following properties. These properties can cause
For more information, see Systemd unit file properties.splunkd
to fail on restart.
RemainAfterExit=yes
ExecStop
- Start
splunkd
as asystemd
service.[sudo] systemctl start Splunkd.service
- Verify that
splunkd
is running as asystemd
service. For example:$SPLUNK_HOME/bin/splunk status splunkd is running (PID: 24772). splunk helpers are running (PIDs: 24843 24857 24984 25032).
Alternatively, you can use
systemctl status <unit_file_name>
to check if thesplunkd
process is running, however you might experience a brief time lag during whichsystemctl status
shows "active" andsplunk status
shows "splunkd is not running".Configuring
systemd
to managesplunkd
as a service creates CPU and Memory cgroups in these locations:
CPU:/sys/fs/cgroup/cpu/system.slice/Splunkd.service
Memory:/sys/fs/cgroup/memory/system.slice/Splunkd.service
- For distributed deployments, repeat steps 1-7 on all search heads and indexers.
Additional options for enable boot-start
The enable boot-start
command supports these additional options:
Install splunk init script
In version 7.2.2 and later, the enable boot-start
command adds a -systemd-managed 0|1
option that controls whether to install the splunk
init
script in /etc/init.d
or the Splunkd.service
unit file in /etc/systemd/system
.
To install the splunk
init
script, specify -systemd-managed 0
:
$SPLUNK_HOME/bin/splunk enable boot-start -systemd-managed 0 -user <username> Init script installed at /etc/init.d/splunk. Init script is configured to run at boot.
See Configure Splunk Enterprise to start at boot time.
In version 7.2.2 through 7.2.x, if you do not specify the -systemd-managed
option, the enable boot-start
command defaults to -systemd-managed 1
and installs the Splunkd.service
unit file.
Specify the unit file name
The default splunkd
unit file name is Splunkd.service
. You can specify a different name for the unit file and update the SPLUNK_SERVER_NAME
value in splunk-launch.conf
using the -systemd-unit-file-name
option. For example, to create a unit file with the name "splunk.service":
$SPLUNK_HOME/bin/splunk enable boot-start -systemd-unit-file-name splunk Overwriting present value (Splunkd) of 'SPLUNK_SERVER_NAME' in /opt/splunk/etc/splunk-launch.conf Init script installed at /etc/systemd/system. Init script is configured to run at boot.
For more information, see Unit file naming considerations.
Manage clusters under systemd
When managing an indexer cluster under systemd
:
- You must use the
sudo
command to start, stop, and restart the cluster master or individual peer nodes usingsystemctl start|stop|restart
commands. - You do not need
sudo
to perform a rolling restart using thesplunk rolling-restart cluster-peers
command, or to take a peer offline using thesplunk offline
command.
When managing a search head cluster under systemd
:
- You must use the
sudo
command to start, stop, and restart cluster members usingsystemctl start|stop|restart
commands. - You do not need
sudo
to perform a rolling restart using thesplunk rolling-restart shcluster-members
command, or to remove a cluster member using thesplunk remove shcluster-members
command.
Configure systemd manually
For instructions on how to manually configure systemd
to run splunkd
as a service, see Configure systemd manually in the Workload management manual.
Configure Splunk Enterprise to start at boot time | Install your license |
This documentation applies to the following versions of Splunk® Enterprise: 7.2.2, 7.2.3, 7.2.4, 7.2.5, 7.2.6, 7.2.7, 7.2.8, 7.2.9, 7.2.10
Feedback submitted, thanks!