Linux kernel memory overcommitting and Splunk crashes
Many distributions of Linux have a parameter that controls how the kernel handles requests for large amounts of memory. The Memory Overcommit tunable kernel parameter (
vm.overcommit_memory) determines whether the kernel accepts or denies such requests.
The parameter can be set by running a command such as
sysctl or by echoing a value to a specific file on the machine. The parameter takes three values:
|0||X||Heuristic overcommit handling, denying blatantly invalid requests|
|1||No overcommit handling|
|2||Denies requests that are equal to or greater than the sum of available swap space and the percentage of physical memory as specified by the |
How this setting impacts Splunk performance
Setting this parameter to anything other than the default of 0 can cause Splunk Enterprise to crash frequently with logs that do not reliably provide information on the root cause of the crash. In particular, setting the
overcommit_memory parameter to 2 can result in crashes that provide no information, which makes troubleshooting and problem diagnosis difficult to impossible.
Confirm that you have set the
vm.overcommit_memory parameter to its default of 0, unless you run an app on the same machine that requires changing this parameter. As a best practice, do not run other applications on a machine that runs Splunk Enterprise, unless those applications are directly related to or required by Splunk services.
See the following for additional information about the memory overcommitting feature:
- Capacity Tuning on the Red Hat Linux Performance Tuning Guide
- Your Linux distribution documentation on memory management
Transparent huge memory pages and Splunk performance
This documentation applies to the following versions of Splunk® Enterprise: 6.5.0, 6.5.1, 6.5.2, 6.5.3, 6.5.4, 6.5.5, 6.5.6, 6.5.7, 6.5.8, 6.5.9, 6.5.10, 6.6.0, 6.6.1, 6.6.2, 6.6.3, 6.6.4, 6.6.5, 6.6.6, 6.6.7, 6.6.8, 6.6.9, 6.6.10, 6.6.11, 6.6.12, 7.0.0, 7.0.1, 7.0.2, 7.0.3, 7.0.4, 7.0.5, 7.0.6, 7.0.7, 7.0.8, 7.0.9, 7.0.10, 7.0.11, 7.1.0, 7.1.1, 7.1.2, 7.1.3, 7.1.4, 7.1.5, 7.1.6, 7.1.7, 7.1.8, 7.1.9, 7.2.0, 7.2.1, 7.2.2, 7.2.3, 7.2.4, 7.2.5, 7.2.6, 7.2.7, 7.2.8, 7.3.0, 7.3.1, 7.3.2