I get errors about ulimit in splunkd.log

Problem

Are you seeing messages about ulimits in splunkd.log while running Splunk software on *nix? Are those hosts experiencing a Splunk software crash?

03-03-2011 21:50:09.027 INFO  ulimit - Limit: virtual address space size: unlimited
03-03-2011 21:50:09.027 INFO  ulimit - Limit: data segment size: 1879048192 bytes [hard maximum: unlimited]
03-03-2011 21:50:09.027 INFO  ulimit - Limit: resident memory size: 2147482624 bytes [hard maximum: unlimited]
03-03-2011 21:50:09.027 INFO  ulimit - Limit: stack size: 33554432 bytes [hard maximum: 2147483646 bytes]
03-03-2011 21:50:09.027 INFO  ulimit - Limit: core file size: 1073741312 bytes [hard maximum: unlimited]
03-03-2011 21:50:09.027 INFO  ulimit - Limit: data file size: 2147483646 bytes
03-03-2011 21:50:09.027 ERROR ulimit - Splunk may not work due to low file size limit
03-03-2011 21:50:09.027 INFO  ulimit - Limit: open files: 1024
03-03-2011 21:50:09.027 INFO  ulimit - Limit: cpu time: unlimited
03-03-2011 21:50:09.029 INFO  loader - Splunkd starting (build 95063).

If so, you might need to adjust your server ulimit settings. The ulimits control the resources available to a *nix shell and any processes that shell starts. A *nix host running Splunk software often needs a higher ulimit setting than the defaults.

Causes

First, determine the ulimit settings on the host that has an issue. There are a few ways you can check your ulimit settings.

• Each time the Splunk Enterprise service is started or restarted, it will report on the ulimits. You can search the internal logs for the report using:

  index=_internal source=*splunkd.log ulimit

  .
• On the *nix command line, find the main Splunk Enterprise process (splunkd) and check the ulimits for that process by running prlimit --pid=$pid
• The monitoring console includes a health check for ulimits. See Access and customize health check in Monitoring Splunk Enterprise.

Next, compare the ulimits reported to the guidance available for Splunk Enterprise.

• For the minimum ulimit setting recommendations, see Considerations regarding system-wide resource limits on *nix systems in the Installation Manual.
• Determine if your organization has internal guidance on ulimits settings for *nix hosts. And if there is, what ulimit values are set and how the settings maintained and distributed across the *nix hosts.

Finally, plan any necessary updates to the ulimit settings using the preferred change management option, and document the changes in a change control.

Solutions

You can set new ulimit values by updating the *nix host configuration files.

• The methods of setting ulimit values will vary based on the *nix distribution and the distribution's service management tool. Review the distribution documentation before updating the ulimits.
• You will need to work with your system administrator to set persistent ulimit values.

Set limits using /etc/security/limits.conf

These instructions are for *nix distributions that utilize the init service.

1. Become the root user or an administrative equivalent with su:
   

   sudo su -
   

2. Open /etc/security/limits.conf with a text editor.
3. Add or change the values in the file. Example:
   

   *  hard  nofile   64000
   *  hard  nproc   16000
   *  hard  fsize     -1  
   

4. Save the file and exit the text editor.
5. Restart the machine to complete the changes.

Set limits using the /etc/systemd configuration files

The *nix distributions that utilize the systemd service can have ulimits set at the system level, user level, and service level.

• The /etc/systemd/system.conf file sets system-wide limits.
• The /etc/systemd/user.conf file sets limits for services that run under a specific user.
• The /etc/systemd/system/Splunkd.service file is a service-level configuration that can be created during the Splunk Software installation process or manually. See Configure systemd using enable boot-start in the Admin Manual.

1. Become the root user or an administrative equivalent with su:
   

   sudo su -
   

2. Open the configuration file with a text editor. Example: /etc/systemd/system/Splunkd.service
3. Add or change the values in the file. Example:
   

   LimitNOFILE=65536
   LimitNPROC=16000
   LimitDATA=8589934592
   LimitFSIZE=infinity
   TasksMax=8192
   

4. Save the file and exit the text editor.
5. Restart the services or the machine to complete the changes.

The latest releases of systemd no longer accept DefaultLimitFSIZE=-1 as a valid parameter. Use DefaultLimitFSIZE=infinity instead.