Linux kernel memory overcommitting and Splunk crashes
Many distributions of Linux have a parameter that controls how the kernel handles requests for large amounts of memory. The Memory Overcommit tunable kernel parameter (vm.overcommit_memory
) determines whether the kernel accepts or denies such requests.
The parameter can be set by running a command such as sysctl
or by echoing a value to a specific file on the machine. The parameter takes three values:
Value | Default | Meaning |
---|---|---|
0 | X | Heuristic overcommit handling, denying blatantly invalid requests |
1 | No overcommit handling | |
2 | Denies requests that are equal to or greater than the sum of available swap space and the percentage of physical memory as specified by the overcommit_ratio parameter
|
How this setting impacts Splunk performance
Setting this parameter to anything other than the default of 0 can cause Splunk Enterprise to crash frequently with logs that do not reliably provide information on the root cause of the crash. In particular, setting the overcommit_memory
parameter to 2 can result in crashes that provide no information, which makes troubleshooting and problem diagnosis difficult to impossible.
Confirm that you have set the vm.overcommit_memory
parameter to its default of 0, unless you run an app on the same machine that requires changing this parameter. As a best practice, do not run other applications on a machine that runs Splunk Enterprise, unless those applications are directly related to or required by Splunk services.
See the following for additional information about the memory overcommitting feature:
- Capacity Tuning on the Red Hat Linux Performance Tuning Guide
- Your Linux distribution documentation on memory management
Transparent huge memory pages and Splunk performance | Splunk Enterprise and NUMA architectures |
This documentation applies to the following versions of Splunk® Enterprise: 7.0.0, 7.0.1, 7.0.2, 7.0.3, 7.0.4, 7.0.5, 7.0.6, 7.0.7, 7.0.8, 7.0.9, 7.0.10, 7.0.11, 7.0.13, 7.1.0, 7.1.1, 7.1.2, 7.1.3, 7.1.4, 7.1.5, 7.1.6, 7.1.7, 7.1.8, 7.1.9, 7.1.10, 7.2.0, 7.2.1, 7.2.2, 7.2.3, 7.2.4, 7.2.5, 7.2.6, 7.2.7, 7.2.8, 7.2.9, 7.2.10, 7.3.0, 7.3.1, 7.3.2, 7.3.3, 7.3.4, 7.3.5, 7.3.6, 7.3.7, 7.3.8, 7.3.9, 8.0.0, 8.0.1, 8.0.2, 8.0.3, 8.0.4, 8.0.5, 8.0.6, 8.0.7, 8.0.8, 8.0.9, 8.0.10, 8.1.0, 8.1.1, 8.1.2, 8.1.3, 8.1.4, 8.1.5, 8.1.6, 8.1.7, 8.1.8, 8.1.9, 8.1.10, 8.1.11, 8.1.12, 8.1.13, 8.1.14, 8.2.0, 8.2.1, 8.2.2, 8.2.3, 8.2.4, 8.2.5, 8.2.6, 8.2.7, 8.2.8, 8.2.9, 8.2.10, 8.2.11, 8.2.12, 9.0.0, 9.0.1, 9.0.2, 9.0.3, 9.0.4, 9.0.5, 9.0.6, 9.0.7, 9.0.8, 9.0.9, 9.0.10, 9.1.0, 9.1.1, 9.1.2, 9.1.3, 9.1.4, 9.1.5, 9.2.0, 9.2.1, 9.2.2, 9.3.0
Feedback submitted, thanks!