Splunk Enterprise and NUMA architectures

CPU architectures featuring non-uniform memory access (NUMA) are increasingly popular, and many servers now offer them: multi-socket systems as well as single-socket, cluster-on-die CPUs. Software that runs on systems with NUMA, but is not optimized for NUMA architectures, can experience performance cliffs as the operating system migrates processes between different CPUs.

Splunk Enterprise is not optimized to run on NUMA architectures. Splunk has observed CPU usage increases of the order of 50% when running Splunk Enterprise on servers with 8 NUMA nodes, and eventual blockage of ingestion in periods of high search load, when the number of active processes on the server puts pressure on the operating system to migrate processes between CPUs more frequently.

Turn off NUMA in any system on which you run Splunk Enterprise software. Check the documentation of your operating system for details on how to do so.