Troubleshoot lookups in Splunk Enterprise Security
Troubleshoot Splunk issues regarding lookups and available memory.
Increasing max_memtable_bytes
When increasing max_memtable_bytes
in the limits.conf file, note that this controls the maximum size for a lookup to be indexed in memory. This means that every time a search runs, it is first indexed, and then loaded into memory. Indexing can impact performance as the size of the lookup grows larger. Smaller and denser lookups perform better in memory, while larger and sparser lookups perform better on disk. 25MB is the default for on-premises and 100MB is the default for cloud. This setting is adjustable, but do not baselessly set the value as big as your biggest lookup without testing and tuning.
Lookups not respecting ASCII name order
Splunk Enterprise does not honor lexicographical order of automatic search-time lookups when some of the lookups in a set are configured to execute in-memory versus when some of the lookups in the set are configured to be indexed.
For instance, if you have max_memtable_bytes
set to 50MB, assets_by_cidr
lookup set to 25MB, and assets_by_str
lookup set to 75MB. This would cause assets_by_str
to be indexed and assets_by_cidr
to run in memory, resulting in assets_by_cidr
inadvertently executing prior to assets_by_str
.
Increase the max_memtable_bytes
of the lookup
stanza in $SPLUNK_HOME/etc/system/default/limits.conf
. See limits.conf in the Splunk Enterprise Admin Manual.
Lookup files growing in excess of 1GB
Lookup table files involved in special search matches, such as CIDR or Wildcard, are required to run in memory. This can lead to running out of memory when using these features.
Increase the max_memtable_bytes
of the lookup
stanza in $SPLUNK_HOME/etc/system/default/limits.conf
. See limits.conf in the Splunk Enterprise Admin Manual.
Increasing max_content_length
When increasing httpServer:max_content_length
in the server.conf file, note that this setting exists to avoid allocating an unreasonable amount of memory from web requests.
Lookup tables exceeding the maximum length
Lookup table files that exceed the HTTP httpServer:max_content_length
in the server.conf file will not be replicated across search head cluster members.
Increase the max_content_length
of the http_input
stanza in $SPLUNK_HOME/etc/system/default/server.conf
. See server.conf in the Splunk Enterprise Admin Manual.
Troubleshoot dashboards in Splunk Enterprise Security | Troubleshoot missing notable events in Splunk Enterprise Security |
This documentation applies to the following versions of Splunk® Enterprise Security: 5.3.1, 6.0.0, 6.0.1, 6.0.2, 6.1.0, 6.1.1, 6.2.0, 6.3.0 Cloud only, 6.4.0, 6.4.1, 6.5.0 Cloud only, 6.5.1 Cloud only, 6.6.0, 6.6.2
Feedback submitted, thanks!