Troubleshoot lookups in Splunk Enterprise Security

Troubleshoot Splunk issues regarding lookups and available memory.

Increasing max_memtable_bytes

When increasing max_memtable_bytes in the limits.conf file, note that this controls the maximum size for a lookup to be indexed in memory. This means that every time a search runs, it is first indexed, and then loaded into memory. Indexing can impact performance as the size of the lookup grows larger. Smaller and denser lookups perform better in memory, while larger and sparser lookups perform better on disk. 25MB is the default for on-premises and 100MB is the default for cloud. This setting is adjustable, but do not baselessly set the value as big as your biggest lookup without testing and tuning.

Lookups not respecting ASCII name order

Splunk Enterprise does not honor lexicographical order of automatic search-time lookups when some of the lookups in a set are configured to execute in-memory versus when some of the lookups in the set are configured to be indexed.

For instance, if you have max_memtable_bytes set to 50MB, assets_by_cidr lookup set to 25MB, and assets_by_str lookup set to 75MB. This would cause assets_by_str to be indexed and assets_by_cidr to run in memory, resulting in assets_by_cidr inadvertently executing prior to assets_by_str.

On the standalone search head or search peers and indexers, configure the setting enforce_auto_lookup_order = true in the [lookup] stanza of the limits.conf configuration file so that the lookup names in the props.conf file are looked up in ASCII order by name. This is the preferred method for the following Splunk Enterprise versions:

• 8.1.5 and higher
• 8.2.3 and higher
• 9.0.0 and higher
• 8.2.2106 and higher

Alternatively, you can increase the max_memtable_bytes of the lookup stanza in $SPLUNK_HOME/etc/system/default/limits.conf.

For more information, see limits.conf configuration file in the Splunk Enterprise Administrator Manual.

Lookup files growing in excess of 1GB

Lookup table files involved in special search matches, such as CIDR or Wildcard, are required to run in memory. This can lead to running out of memory when using these features.

Increase the max_memtable_bytes of the lookup stanza in $SPLUNK_HOME/etc/system/default/limits.conf. See limits.conf in the Splunk Enterprise Admin Manual.

Increasing max_content_length

When increasing httpServer:max_content_length in the server.conf file, note that this setting exists to avoid allocating an unreasonable amount of memory from web requests.

Lookup tables exceeding the maximum length

Lookup table files that exceed the HTTP httpServer:max_content_length in the server.conf file will not be replicated across search head cluster members.

Increase the max_content_length of the http_input stanza in $SPLUNK_HOME/etc/system/default/server.conf. See server.conf in the Splunk Enterprise Admin Manual.