Troubleshoot performance issues cause by searches and lookups in Splunk Enterprise Security

Issue

Performance issues caused due to excessive memory usage by lookups or searches.

Causes

1. Indexing a search or a large lookup consumes excessive memory space: Indexing can impact performance as the size of the lookup grows larger. Smaller and denser lookups perform better in memory, while larger and sparser lookups perform better on disk. 25MB is the default for on-premises and 100MB is the default for cloud.

2. Lookup files are larger in size such as over 1GB: Lookup table files involved in special search matches, such as CIDR or Wildcard, are required to run in memory. This can lead to running out of memory when using these features.

3. Lookups do not follow the ASCII name order: Splunk Enterprise does not honor the lexicographical order of automatic search-time lookups when some of the lookups in a set are configured to run in-memory versus when some of the lookups in the set are configured to be indexed.

For instance, if you have max_memtable_bytes set to 50MB, assets_by_cidr lookup set to 25MB, and assets_by_str lookup set to 75MB. This would cause assets_by_str to be indexed and assets_by_cidr to run in memory, resulting in assets_by_cidr to inadvertently run prior to assets_by_str.

Solutions

1. Control the maximum size of a lookup that can be indexed in memory: Increase the max_memtable_bytes in the $SPLUNK_HOME/etc/system/default/limits.conf configuration file. Thus, every time a search runs, it is first indexed, and then loaded into memory.

Though this setting is adjustable, you mustn't set the value as big as your biggest lookup without testing and tuning.

2. Increase the max_content_length setting: Increase the max_content_length of the http_input stanza in $SPLUNK_HOME/etc/system/default/server.conf.

When increasing httpServer:max_content_length in the server.conf configuration file, note that this setting exists to avoid allocating an unreasonable amount of memory from web requests.

Lookup table files that exceed the HTTP httpServer:max_content_length in the server.conf configuration file are not replicated across search head cluster members.

3. Configure the setting enforce_auto_lookup_order = true: Configure this setting in the [lookup] stanza of the limits.conf configuration file on the standalone search head or search peers and indexers so that the lookup names in the props.conf file are looked up in ASCII order by name.

This is the preferred method for the following Splunk Enterprise versions:

• 8.1.5 and higher
• 8.2.3 and higher
• 9.0.0 and higher
• 8.2.2106 and higher

See also

For more information on configuration files, see the product documentation:

• limits.conf configuration file in the Splunk Enterprise Admin Manual.
• server.conf configuration file in the Splunk Enterprise Admin Manual.