I get errors about ulimit in splunkd.log
Are you seeing messages like these in splunkd.log while running Splunk software on *nix, possibly accompanied by a Splunk software crash?
03-03-2011 21:50:09.027 INFO ulimit - Limit: virtual address space size: unlimited 03-03-2011 21:50:09.027 INFO ulimit - Limit: data segment size: 1879048192 bytes [hard maximum: unlimited] 03-03-2011 21:50:09.027 INFO ulimit - Limit: resident memory size: 2147482624 bytes [hard maximum: unlimited] 03-03-2011 21:50:09.027 INFO ulimit - Limit: stack size: 33554432 bytes [hard maximum: 2147483646 bytes] 03-03-2011 21:50:09.027 INFO ulimit - Limit: core file size: 1073741312 bytes [hard maximum: unlimited] 03-03-2011 21:50:09.027 INFO ulimit - Limit: data file size: 2147483646 bytes 03-03-2011 21:50:09.027 ERROR ulimit - Splunk may not work due to low file size limit 03-03-2011 21:50:09.027 INFO ulimit - Limit: open files: 1024 03-03-2011 21:50:09.027 INFO ulimit - Limit: cpu time: unlimited 03-03-2011 21:50:09.029 INFO loader - Splunkd starting (build 95063).
If so, you might need to adjust your server ulimit. Ulimit controls the resources available to a *nix shell and processors the *nix shell has started. A machine running Splunk software needs higher limits than are provided by default.
To check your limits, type:
Or restart Splunk Enterprise and look in splunkd.log for events mentioning ulimit:
index=_internal source=*splunkd.log ulimit
You probably want your new values to stay set even after you reboot. To persistently modify the values, edit settings in
The most important values are:
- The file size (
ulimit -f). The size of an uncompressed bucket file can be very high.
- The data segment size (
ulimit -d). Increase the value to at least 1 GB = 1073741824 bytes.
- The number of open files (
ulimit -n), sometimes called the number of file descriptors. Increase the value to at least 8192 (depending on your server capacity).
- The max user processes (
ulimit -u). Increase to match the file descriptors. This limit is important for the number of http threads.
Another value that you might need to modify on an older system (but not on most modern systems) is the system-wide file size,
Why must you increase ulimit to run Splunk software? Well, you might concurrently need file descriptors for every forwarder socket and every deployment client socket. Each bucket can use 10 to 100 files, every search consumes up to 3, and then consider every file to be indexed and every user connected.
What do I do with buckets?
Common issues with Splunk and WMI
This documentation applies to the following versions of Splunk® Enterprise: 4.3, 4.3.1, 4.3.2, 4.3.3, 4.3.4, 4.3.5, 4.3.6, 4.3.7, 5.0, 5.0.1, 5.0.2, 5.0.3, 5.0.4, 5.0.5, 5.0.6, 5.0.7, 5.0.8, 5.0.9, 5.0.10, 5.0.11, 5.0.12, 5.0.13, 5.0.14, 5.0.15, 5.0.16, 6.0, 6.0.1, 6.0.2, 6.0.3, 6.0.4, 6.0.5, 6.0.6, 6.0.7, 6.0.8, 6.0.9, 6.0.10, 6.0.11, 6.0.12, 6.1, 6.1.1, 6.1.2, 6.1.3, 6.1.4, 6.1.5, 6.1.6, 6.1.7, 6.1.8, 6.1.9, 6.1.10, 6.1.11