Configure the SmartStore cache manager
The cache manager maximizes search efficiency through intelligent management of the local cache. It favors retaining in the cache copies of buckets and files that have a high likelihood of participating in future searches. When the cache fills up, the cache manager removes, or "evicts", copies of buckets that are least likely to participate in future searches.
Since the cache manager removes only the cached copies of buckets, the eviction process does not result in loss of data. The master copies continue to reside in remote storage,
For details on how the cache manager operates, see The SmartStore cache manager.
Cache manager settings reside in the
[cachemanager] stanza in server.conf. In the case of an indexer cluster, you configure the cache manager on each peer node.
The cache manager operates at the global level, across all indexes on an indexer. Aside from the recency settings, you cannot configure the cache manager on a per-index basis.
Set the cache eviction policy
eviction_policy setting in
server.conf determines the cache eviction policy.
|lru (default)||Evict the least recently used bucket.|
|clock||Evict the bucket with the oldest events first, unless it has been accessed recently.|
|lrlt||Evict the bucket with the oldest events first.|
|random||Randomly evict a bucket.|
Do not change
eviction_policy from its default value of "lru" without consulting Splunk Support.
Initiate eviction based on occupancy of the cache's disk partition
These settings in
server.conf initiate eviction based on occupancy of the cache's disk partition:
max_cache_sizesetting specifies the maximum occupied space, in megabytes, for the disk partition that contains the cache.
minFreeSpacesetting specifies the minimum free space, in megabytes, for a partition.
eviction_paddingsetting controls the amount of additional space, in megabytes, that the cache manager protects, beyond the
minFreeSpace setting is not strictly a cache-specific setting, and therefore it does not reside in the
[cachemanager] stanza, but it nevertheless helps determine cache size limits.
When the occupied space on the cache's partition exceeds
max_cache_size, or the partition's free space falls below (
eviction_padding), the cache manager begins to evict data.
Set cache retention periods based on data recency
You can protect recently indexed data from eviction. You can use this capability in two ways:
- On a global level (across all indexes), to favor recently indexed data over recently used data.
- On a per-index level, to favor data in critical indexes over data in non-critical indexes.
To set cache retention periods based on data recency, use the
hotlist_bloom_filter_recency_hours settings. These settings serve to override the eviction policy. You can scope these settings globally or on a per-index level.
hotlist_recency_secs setting causes the cache manager to protect buckets that contain recent data over other buckets. When eviction is necessary, the cache manager will not evict buckets until their configured retention periods have passed, unless all other buckets have already been evicted.
hotlist_bloom_filter_recency_hours setting protects certain small metadata files, such as the
bloomfilter file, from eviction. By inspecting such metadata files, the cache manager can sometimes eliminate the need to fetch larger bucket files, such as the rawdata journal and the
tsidx files, from remote storage when handling search requests. See The SmartStore cache manager.
When you configure these settings globally, they override the eviction policy, which, by default, favors buckets that have been recently searched. For example, if
hotlist_recency_secs is set globally to 604800 (7 days), the cache manager will attempt to retain buckets with data that is less than seven days old. It will instead evict older buckets, even if those older buckets were searched more recently. The cache manager will only evict buckets containing data less than seven days old if there are no older buckets to evict.
By configuring the recency settings on a per-index level, you can favor data in critical indexes over data in less critical indexes. Since all SmartStore indexes share the cache and otherwise follow the global cache eviction policy, the per-index recency settings provide the only means to retain data from critical indexes for a longer period than data from less critical indexes.
For example, if you have an index with critical data, such as the ES
threat_activity index, and another index whose data is less critical, such as the default
_internal index, you can set
hotlist_recency_secs to 5184000 (60 days) for
threat_activity, while keeping the default setting of 86400 (1 day) for
_internal. By doing so, you cause the cache manager to favor
threat_activity buckets over
_internal buckets, thus reducing the likelihood that the cache will need to fetch data from the remote store to handle
Details of settings that control retention based on data recency
hotlist_recency_secs setting affects the cache retention period for warm buckets. The cache manager attempts to defer bucket eviction until the interval between the bucket's latest time and the current time exceeds this setting. This setting defaults to 86400 seconds, or 24 hours.
hotlist_bloom_filter_recency_hours setting affects the cache retention period for small warm bucket files. The cache manager attempts to defer eviction of the non-journal and non-tsidx bucket files, such as the
bloomfilter file, until the interval between the bucket's latest time and the current time exceeds this setting. This setting defaults to 360 hours, or 15 days.
You configure each setting in
indexes.conf, depending on whether you want to configure the setting for all SmartStore indexes or for individual indexes.
Configure recency for all indexes
To configure the
hotlist_bloom_filter_recency_hours settings globally, for all SmartStore indexes, you must set them in the
[cachemanager] stanza in
You can override the global settings on a per-index basis.
Configure recency for individual indexes
To configure the
hotlist_bloom_filter_recency_hours settings on a per-index basis, you must set them in each index's stanza in
If you do not configure the settings for a particular SmartStore index, that index inherits the global value from
Set the maximum download and upload rates
max_concurrent_downloads setting in
server.conf specifies the maximum number of buckets that can be downloaded simultaneously from remote storage. Its default is 8.
max_concurrent_uploads setting in
server.conf specifies the maximum number of buckets that can be uploaded simultaneously to remote storage. Its default is 8.
Configure data retention for SmartStore indexes
This documentation applies to the following versions of Splunk® Enterprise: 7.2.0, 7.2.1, 7.2.2, 7.2.3, 7.2.4, 7.2.5, 7.2.6, 7.2.7, 7.2.8, 7.2.9, 7.2.10, 7.3.0, 7.3.1, 7.3.2, 7.3.3, 7.3.4, 7.3.5, 7.3.6, 7.3.7, 7.3.8