Splunk® Enterprise

Search Manual

Acrobat logo Download manual as PDF


Splunk Enterprise version 6.x is no longer supported as of October 23, 2019. See the Splunk Software Support Policy for details. For information about upgrading to a supported version, see How to upgrade Splunk Enterprise.
Acrobat logo Download topic as PDF

Limit search process memory usage

Splunk software can be configured to automatically terminate search job processes that exceed a threshold of a configured quantity of resident memory in use.

You might be interested in using this feature if:

  • You want to be proactive and avoid a scenario where one runaway search causes one or several of your search peers to crash.
  • You already have encountered this scenario and do not want it to happen again.
  • In the Distributed Management Console, the Search Activity: Instance view exposes one or more searches that consume dangerous amounts of physical memory. You can see this information in the Top 10 memory-consuming search panel.

If you have Splunk Cloud and want to adjust this threshold, you must file a Support ticket, because you do not have access to the limits.conf file.

What does this threshold do?

Enabling this threshold limits the maximum memory permitted for each search process. A search process that is an outlier in memory size is automatically killed off, limiting damage.

This threshold uses process resource usage information that is recorded by platform instrumentation. So this feature works only on *nix, Solaris, and Windows platforms.

Search memory is checked periodically, so a rapid spike might exceed the configured limit.

The functionality is wired into the DispatchDirectoryReaper, so stalls in the reaper components also cause stalls in how often the memory of searches are checked.

Enable a search process memory threshold

The search process memory tracking is disabled by default.

1. See How to edit a configuration file in the Admin Manual.

2. Open the limits.conf file.

3. In the [search] stanza, change the setting for the enable_memory_tracker attribute to true.

4. Review and adjust the memory limit.

You can set the limit to an absolute amount or a percentage of the identified system maximum, using search_process_memory_usage_threshold or search_process_memory_usage_percentage_threshold, respectively. Searches are always tested against both values, and the lower value applies. See limits.conf.spec in the Admin Manual.

5. To enable the configuration changes, restart Splunk Enterprise.

Where is threshold activity logged?

If the threshold causes a search process to be stopped on a search head, an error is inserted into the search artifact file info.csv. If the search is run through Splunk Web, this error message also appears in Splunk Web. The error states that the process was terminated and specifies the limit setting and value.

If the threshold causes a search process to be stopped on a search peer, a WARN message is logged in the splunkd.log file in the StreamedSearch category.

In both cases, a WARN message is logged in the splunkd.log file in the DispatchReaper category.

The messages are similar to: 

Forcefully terminated search process with sid=... since 
its \[relative physical or physical] memory usage (... \[MB or %]) 
has exceeded the \[relative physical or physical] memory 
threshold specified in limits.conf/...setting name.. (...setting value...)
Last modified on 18 June, 2020
PREVIOUS
Dispatch directory and search artifacts 		  NEXT
Manage Splunk Enterprise jobs from the OS

This documentation applies to the following versions of Splunk® Enterprise: 6.3.0, 6.3.1, 6.3.2, 6.3.3, 6.3.4, 6.3.5, 6.3.6, 6.3.7, 6.3.8, 6.3.9, 6.3.10, 6.3.11, 6.3.12, 6.3.13, 6.3.14, 6.4.0, 6.4.1, 6.4.2, 6.4.3, 6.4.4, 6.4.5, 6.4.6, 6.4.7, 6.4.8, 6.4.9, 6.4.10, 6.4.11, 6.5.0, 6.5.1, 6.5.2, 6.5.3, 6.5.4, 6.5.5, 6.5.6, 6.5.7, 6.5.8, 6.5.9, 6.5.10, 6.6.0, 6.6.1, 6.6.2, 6.6.3, 6.6.4, 6.6.5, 6.6.6, 6.6.7, 6.6.8, 6.6.9, 6.6.10, 6.6.11, 6.6.12, 7.0.0, 7.0.1, 7.0.3, 7.0.5, 7.0.8, 7.0.9, 7.0.10, 7.0.11, 7.0.13, 7.1.0, 7.1.1, 7.1.2, 7.1.3, 7.1.4, 7.1.5, 7.1.6, 7.1.7, 7.1.8, 7.1.9, 7.1.10, 7.2.0, 7.2.1, 7.2.2, 7.2.3, 7.2.4, 7.2.5, 7.2.6, 7.2.7, 7.2.8, 7.2.9, 7.2.10, 7.3.0, 7.3.1, 7.3.2, 7.3.3, 7.3.4, 7.3.5, 7.3.6, 7.3.7, 7.3.8, 7.3.9, 8.0.0, 8.0.1, 8.0.2, 8.0.3, 8.0.4, 8.0.5, 8.0.6, 8.0.7, 8.0.8, 8.1.0, 8.1.1, 8.1.2, 7.0.2, 7.0.4, 7.0.6, 7.0.7

Was this documentation topic helpful?

You must be logged into splunk.com in order to post comments. Log in now.

Please try to keep this discussion focused on the content covered in this documentation topic. If you have a more general question about Splunk functionality or are experiencing a difficulty with Splunk, consider posting a question to Splunkbase Answers.

0 out of 1000 Characters