Splunk® Enterprise

Admin Manual

Download manual as PDF

Download topic as PDF

Run Splunk Enterprise as a systemd service

Splunk Enterprise 7.2.2 and later adds broad support for systemd on Linux with an updated enable boot-start command that lets you automatically configure systemd to manage splunkd as a service.

What is systemd?

systemd is a system startup and service manager that is widely deployed as the default init system on most major Linux distributions. You can configure systemd to manage processes, such as splunkd, as services, and allocate system resources to those processes under cgroups.

systemd advantages

systemd offers the following general advantages:

  • Enhanced parallel processing.
  • Simplified configuration with standardized unit text files. No scripts required.
  • Improved mechanism for expressing dependencies. For example, you can specify in the unit file that the network must be up before startup of the splunkd service occurs.

systemd offers these additional specific advantages for Splunk deployments:

  • Start splunkd at boot.
  • Monitor and manage splunkd service during runtime.
  • Provides tools to debug and troubleshoot boot-time and service activities.
  • Allows more control over plug-in monitoring tools that track the status of Splunk instances.
  • Simplifies the set up of cgroups required for workload management in Splunk Enterprise. See Set up Linux for workload management.

Configure systemd to manage splunkd

There are two ways to configure systemd to manage splunkd as a service:

System requirements

  • To run splunkd as a systemd service requires one of the following supported Linux distributions:
    • RHEL 6 and 7
    • CentOS 6 and 7
    • Ubuntu 16.04 LTS and later
    • Suse 12
  • To configure systemd using enable boot-start requires Splunk Enterprise version 7.2.2 or later.
  • To enable workload management in Splunk Enterprise under systemd requires systemd version 219 or higher. For more information, see Linux operating system requirements in the Workload Management manual.

Permissions requirements

The enable boot-start command and systemd have the following permissions requirements:

  • Non-root users must have super user permissions to configure systemd using enable boot-start.
  • Non-root users must have super user permissions to run start, stop, and restart commands under systemd.

For instructions on how to create a new user with super user permissions, see your Linux documentation. The specific steps might vary depending on the specific Linux distribution.

You must use sudo to run splunk start|stop|restart commands. If you do not use sudo, you must authenticate. For example:

==== AUTHENTICATING FOR org.freedesktop.systemd1.manage-units ===
Authentication is required to manage system services or units.
Multiple identities can be used for authentication:
 1.  <username_1>
 2.  <username_2>
Choose identity to authenticate as (1-2): 2
Password: 
==== AUTHENTICATION COMPLETE ===

Configure systemd using enable boot-start

You can configure systemd to manage splunkd as a service using the enable boot-start command.

  1. Log into the machine on which you want to configure systemd to manage splunkd as a service.
  2. Stop splunkd.
    $SPLUNK_HOME/bin/splunk stop
    
  3. If you previously enabled Splunk Enterprise to start at boot using the enable boot-start command, run disable boot-start to remove the splunk init script located in /etc/init.d and its symbolic links.
    [sudo] $SPLUNK_HOME/bin/splunk disable boot-start
    
  4. Run the enable boot-start command to create the splunkd unit file .
    [sudo] $SPLUNK_HOME/bin/splunk enable boot-start -user <username>
    

    This installs the following unit file, named Splunkd.service by default, in /etc/systemd/system. To specify a different unit file name, use the -systemd-unit-file-name option. See Specify the splunkd unit file name.

    #This unit file replaces the traditional start-up script for systemd
    #configurations, and is used when enabling boot-start for Splunk on
    #systemd-based Linux distributions.
    
    [Unit]
    Description=Systemd service file for Splunk, generated by 'splunk enable boot-start'
    After=network.target
    
    [Service]
    Type=simple
    Restart=always
    ExecStart=/opt/splunk/bin/splunk _internal_launch_under_systemd
    LimitNOFILE=65536
    SuccessExitStatus=51 52
    RestartPreventExitStatus=51
    RestartForceExitStatus=52
    User=<username>
    Delegate=true
    MemoryLimit=100G
    CPUShares=1024
    PermissionsStartOnly=true
    ExecStartPost=/bin/bash -c "chown -R <username>:<username> /sys/fs/cgroup/cpu/system.slice/%n"
    ExecStartPost=/bin/bash -c "chown -R <username>:<username> /sys/fs/cgroup/memory/system.slice/%n"
    
    [Install]
    WantedBy=multi-user.target
    

    If you run enable boot-start as root without specifying -user, the default unit file appears as follows:

    [Unit]
    Description=Systemd service file for Splunk, generated by 'splunk enable boot-start'
    After=network.target
    
    [Service]
    Type=simple
    Restart=always
    ExecStart=/opt/splunk/bin/splunk _internal_launch_under_systemd
    LimitNOFILE=65536
    SuccessExitStatus=51 52
    RestartPreventExitStatus=51
    RestartForceExitStatus=52
    Delegate=true
    MemoryLimit=100G
    CPUShares=1024
    
    [Install]
    WantedBy=multi-user.target
    

    These unit file properties must be set as follows. Do not change these values unless you are familiar with systemd or receive guidance from Splunk support.
    Type=simple
    Restart=always
    ExecStart=$SPLUNK_HOME/bin/splunk _internal_launch_under_systemd
    Delegate=true This property is required to enable workload management. See Configure workload managment.

    Do not use the following unit file properties. These properties can cause splunkd to fail on restart.
    KillMode=none
    RemainAfterExit=yes
    ExecStop

    For more information, see Systemd unit file properties.

  5. Start splunkd.
    [sudo] $SPLUNK_HOME/bin/splunk start
    

    This starts splunkd as a systemd service.

    Under systemd, splunk start|stop|restart commands are mapped to systemctl start|stop|restart commands.

  6. Verify that splunkd is running as a systemd service. For example:
    $SPLUNK_HOME/bin/splunk status
    splunkd is running (PID: 24772).
    splunk helpers are running (PIDs: 24843 24857 24984 25032).
    

    You can use systemctl status to check if the splunkd process is running, but there might be a brief lag time during which systemctl status shows "active" and splunk status shows "splunkd is not running".

    When you configure systemd to run splunkdas a service, systemd creates corresponding CPU and Memory cgroups in these locations:

    CPU: /sys/fs/cgroup/cpu/system.slice/<SPLUNK_SERVER_NAME>.service
    Memory: /sys/fs/cgroup/memory/system.slice/<SPLUNK_SERVER_NAME>.service
    
  7. For distributed deployments, repeat steps 1-6 on all search heads and indexers.

Additional options for enable boot-start

The enable boot-start command supports these additional options:

Install splunk init script

In version 7.2.2 and later, the enable boot-start command adds a -systemd-managed 0|1 option that controls whether to install the splunk init script in /etc/init.d or the splunkd unit file in /etc/systemd/system.

To install the splunk init script, specify -systemd-managed 0:

$SPLUNK_HOME/bin/splunk enable boot-start -systemd-managed 0 -user <username>
Init script installed at /etc/init.d/splunk.
Init script is configured to run at boot.

See Configure Splunk Enterprise to start at boot time.

In version 7.2.2 through 7.2.x, if you do not specify the -systemd-managed option, the enable boot-start command defaults to -systemd-managed 1 and installs the splunkd unit file.

Specify the unit file name

The default splunkd unit file name is Splunkd.server. The unit file name is based on the SPLUNK_SERVER_NAME value in splunk-launch.conf, which by default is Splunkd.

You can specify a different name for the splunkd unit file and update the existing SPLUNK_SERVER_NAME value in splunk-launch.conf using the -systemd-unit-file-name option. For example, to create a unit file with the name "splunk.server":

$SPLUNK_HOME/bin/splunk enable boot-start -systemd-unit-file-name splunk
Overwriting present value (Splunkd) of 'SPLUNK_SERVER_NAME' in /opt/splunk/etc/splunk-launch.conf
Init script installed at /etc/systemd/system.
Init script is configured to run at boot.

Manage clusters under systemd

When managing an indexer cluster under systemd:

  • You must use the sudo command to start, stop, and restart the cluster master or individual peer nodes using splunk start|stop|restart commands.
  • You do not need sudo to perform a rolling restart using the splunk rolling-restart cluster-peers command, or to take a peer offline using the splunk offline command.

When managing a search head cluster under systemd:

  • You must use the sudo command to start, stop, and restart cluster members using splunk start|stop|restart commands.
  • You do not need sudo to perform a rolling restart using the splunk rolling-restart shcluster-members command, or to remove a cluster member using the splunk remove shcluster-members command.

Configure systemd manually

For instructions on how to manually configure systemd to run splunkd as a service, see Configure systemd manually in the Workload management manual.

PREVIOUS
Configure Splunk Enterprise to start at boot time
  NEXT
Install your license

This documentation applies to the following versions of Splunk® Enterprise: 7.2.2, 7.2.3, 7.2.4, 7.2.5


Comments

Hi Michael-

Thank you for your comment.

The workaround for this issue is to manually edit and update the unit file with the appropriate group info. In future releases, the unit file will contain a unique <userid>:<groupid> to prevent this issue.

Sroback splunk, Splunker
March 5, 2019

In the "Configure systemd using enable boot-start" section, there are these lines:

ExecStartPost=/bin/bash -c "chown -R <username>:<username> /sys/fs/cgroup/cpu/system.slice/%n"
ExecStartPost=/bin/bash -c "chown -R <username>:<username> /sys/fs/cgroup/memory/system.slice/%n"

Those are created by issuing splunk enable boot-start with the -user option and parameter. However, that is not valid, and those commands are not valid, if a group does not exist on the system with the name of "<username>" (whatever is entered there). This causes the systemctl start command to fail, and Splunk will never start up.

MichaelRye
February 19, 2019

Intermediate -
Thank you for your feedback and posting your workaround for standardizing the service name.

Another possible workaround is to change the default value of SPLUNK_SERVER_NAME in splunk-launch.conf (which is configurable) from “Splunkd” to a name of your choice, such as “splunk”.

HTH

Sroback splunk, Splunker
February 7, 2019

PS to my last: Got sick of the various service names and just created a "splunk.service" symlink to /etc/systemd/system/Splunkd.server OR /etc/systemd/SplunkForwarder.service as appropriate. A quick "systemctl daemon-reload" later an my sanity is preserved. It's a pity I would have preferred our systems not to diverge from the defaults.

Intermediate
February 5, 2019

Thank you for supporting systemd, we've been meaning to do this for some time but prefer to keep our environment as standard as possible.

I've gotta ask, why didn't you call the services "splunk"? Why are they "Splunkd" and "SplunkForwarder". Not only is the use of uppercase unnecessary and un-unix-like but the 'd' after the Splunk server process name isn't mirrored in the Forwader's name, then there's the fact that previously both services were just /etc/init.d/splunk

This is annoying as (to keep our environment standard) we now have to updated all our automation scripts to cater for two different process names and illogical ones at that.

Intermediate
February 5, 2019

Was this documentation topic helpful?

Enter your email address, and someone from the documentation team will respond to you:

Please provide your comments here. Ask a question or make a suggestion.

You must be logged into splunk.com in order to post comments. Log in now.

Please try to keep this discussion focused on the content covered in this documentation topic. If you have a more general question about Splunk functionality or are experiencing a difficulty with Splunk, consider posting a question to Splunkbase Answers.

0 out of 1000 Characters