Limit search process memory usage

Splunk software tracks search process memory and supports a search process memory threshold, which automatically terminates search job processes that exceed a certain amount of resident memory in use.

The search process memory threshold limits the maximum memory permitted for each search process and automatically terminates search processes that are outliers in memory size, which limits damage to other processes. This threshold uses process resource usage information that is recorded by platform instrumentation, and as a result, works only on *nix, Solaris, and Windows platforms. For more information, see:

• Introspection endpoint descriptions in the REST API Reference Manual.
• About the platform instrumentation framework in the Troubleshooting Manual.

Search memory is checked periodically, so a rapid spike might exceed the configured limit. The functionality is built into the DispatchDirectoryReaper, so stalls in the reaper components also cause stalls in how often the memory of searches are checked.

By default, search process memory tracking is turned on for Splunk Cloud Platform, and turned off for Splunk Enterprise.

The search process memory threshold on Splunk Cloud Platform

Search process memory tracking is turned on by default for Splunk Cloud Platform. The following are the default settings for the search process memory threshold on Splunk Cloud Platform:

• The threshold for search process memory usage is set to 80 percent. This is the percentage of total memory that a search process is allowed to consume. Search processes that violate the threshold percentage are terminated.
• The threshold for search process memory usage is set to 0, which means that search processes are allowed to grow unbounded in terms of in memory usage.

The default settings for the search process memory threshold can't be changed on Splunk Cloud Platform.

The search process memory threshold on Splunk Enterprise

You can configure the search process memory threshold on Splunk Enterprise by updating the limits.conf file. You might want to set the search process memory threshold if:

• You want to be proactive and avoid a scenario where one runaway search causes one or several of your search peers to crash.
• You already have encountered this scenario and do not want it to happen again.
• In the Distributed Management Console, the Search Activity: Instance view exposes one or more searches that consume dangerous amounts of physical memory. You can see this information in the Top 10 memory-consuming search panel.

Search process memory tracking is turned off by default on Splunk Enterprise. To turn on the search process memory threshold in the limits.conf file, follow these steps.

Prerequisites

• Only users with file system access, such as system administrators, can edit configuration files.
• Review the steps in How to edit a configuration file in the Splunk Enterprise Admin Manual.

Never change or copy the configuration files in the default directory. The files in the default directory must remain intact and in their original location. Make changes to the files in the local directory.

Steps

1. Open or create a local limits.conf file in the desired path. For example, use the $SPLUNK_HOME/etc/apps/search/local path to apply this change only to the Search app.
2. Under the [search] stanza, change the setting for the enable_memory_tracker setting to true.
3. Review and adjust the memory limit. You can set the limit to an absolute amount or a percentage of the identified system maximum, using search_process_memory_usage_threshold or search_process_memory_usage_percentage_threshold, respectively. Searches are always tested against both values, and the lower value applies. See limits.conf.spec in the Admin Manual.
4. To make the configuration changes take effect, restart Splunk Enterprise.

Where is threshold activity logged?

Threshold activity is logged in different places depending on whether the threshold causes a search process to be stopped on a search head or a search peer.

Search head logging

If the threshold causes a search process to be stopped on a search head, an error is inserted into the search.log and the search artifact file info.csv on the search head. The error message is also displayed in Splunk Web below the search bar and logged in the splunkd.log file in the DispatchReaper category.

The error states that the process was terminated and specifies the limit setting and value. The error message differs depending on whether the physical memory usage (in megabytes) or relative physical memory usage (in percent), or both, exceeded the threshold. The message looks something like this:

The search process with sid=<sid name> was forcefully terminated because both its physical memory usage ( <specified in MB> ) and its relative physical memory usage ( <specified in percent> ) have exceeded the 'search_process_memory_usage_threshold' ( <specified in MB> ) and 'search_process_memory_usage_percentage_threshold' (<specified in percent>) settings in limits.conf.

Search peer logging

If the threshold causes a search process to be stopped on a search peer, an error message is logged in the splunkd.log file in the StreamedSearch category and the splunkd.log file in the DispatchReaper category.

The error states that the process was terminated and specifies the limit setting and value. The error message differs depending on whether the physical memory usage (in megabytes) or relative physical memory usage (in percent), or both, exceeded the threshold. The message looks something like this:

Forcefully terminated search process with sid=<sid name> since both its physical memory usage ( <specified in MB> ) and the relative physical memory usage (<specified in percent>) has exceeded the physical memory thresholds specified in limits.conf / search_process_memory_usage_threshold ( <specified in MB> ) and limits.conf/search_process_memory_usage_percentage_threshold <specified in percent>) respectively.