Splunk Enterprise and NUMA architectures

CPU architectures that feature the non-uniform memory access (NUMA) design are becoming increasingly popular, and many servers now offer them: multi-socket systems as well as single-socket, cluster-on-die CPUs. Software that is not optimized for NUMA can experience performance cliffs on machines with NUMA turned on, as the operating system migrates processes between different CPUs.

Splunk Enterprise is not optimized to run on machines with NUMA turned on. Splunk has observed CPU usage increases of the order of 50% when Splunk Enterprise runs on servers with 8 NUMA nodes, and eventual blockage of ingestion in periods of high search load, when the number of active processes on the server puts pressure on the operating system to migrate processes between CPUs more frequently.

Turn off NUMA on any machine on which you run Splunk Enterprise software. This restriction applies to universal forwarders as well as Splunk Enterprise when you configure it as a heavy forwarder. Check the documentation of your operating system for details on how to do so.