Configure SmartStore

SmartStore configuration settings reside in three files:

• indexes.conf
• server.conf
• limits.conf

SmartStore settings in indexes.conf

The SmartStore settings in indexes.conf enable and control SmartStore indexes.

You can enable SmartStore for all of an indexer's indexes, or you can enable SmartStore on an index-by-index basis, allowing a mix of SmartStore and non-SmartStore indexes on the same indexer.

When you configure these settings on an indexer cluster's peer nodes, you must deploy the settings through the configuration bundle method. As with all settings in indexes.conf, SmartStore settings must be the same across all peer nodes.

The table lists the main SmartStore-related settings in indexes.conf .

SmartStore indexes.conf setting	Stanza level	Description
remotePath = <root path for remote volume>	index or global	Enables SmartStore and sets the remote path for the index's volume. See Deploy SmartStore on a new indexer cluster.
storageType = remote	volume	Sets the volume's storage type to remote. See Deploy SmartStore on a new indexer cluster.
path = <scheme>://<remote-location-specifier>	volume	Sets the remote storage location where indexes reside. Each SmartStore index resides directly below the location specified by the path setting. The <scheme> identifies a supported remote storage system type, such as s3 (S3), gs (GCS), or azure (Azure). The <remote-location-specifier> is a string specific to the remote storage system that specifies the location of the indexes inside the remote system. See Deploy SmartStore on a new indexer cluster. Caution: The value of the path setting for each remote volume stanza must be unique to the indexer cluster or standalone indexer. You can share remote volumes only among indexes within a single cluster or standalone indexer. For example, if indexes on one cluster use a particular remote volume, no index on any other cluster or standalone indexer can use the same remote volume.
maxGlobalDataSizeMB = <integer>	index or global	Determines bucket freezing behavior. Sets the maximum amount of space that the warm and cold buckets of a SmartStore index can occupy. When this maximum is exceeded, the oldest bucket gets frozen. See Configure data retention for SmartStore indexes.
maxGlobalRawDataSizeMB= <integer>	index or global	Determines bucket freezing behavior. Sets the maximum amount of cumulative raw data allowed in the warm and cold buckets of a SmartStore index. When this maximum is exceeded, the oldest bucket gets frozen. See Configure data retention for SmartStore indexes.
hotlist_recency_secs = <integer>	index	Specifies the time period, based on the bucket's age, that the cache manager attempts to protect a recent bucket from eviction. This setting operates on a per-index level, while the version of the setting in server.conf operates across all indexes. See Configure the SmartStore cache manager.
hotlist_bloom_filter_recency_hours = <integer>	index	Specifies the time period, based on the bucket's age, that the cache manager attempts to protect the bucket's non-journal and non-tsidx files, such as the bloomfilter file, from eviction. This setting operates on a per-index level, while the version of the setting in server.conf operates across all indexes. See Configure the SmartStore cache manager.

In indexes.conf, you can also configure various settings specific to your remote storage type. For example, there are a number of settings that begin with remote.s3, such as remote.s3.access_key and remote.s3.secret_key. These settings are specific to S3. Similarly, there are a number of settings that begin with remote.gs and remote.azure. These settings are specific to GCS and Azure, respectively. You configure these settings in the stanza where you configure the remote volume. For details on these and other indexes.conf settings, see indexes.conf.spec.

When specifying a setting that begins with remote.s3, you must use a lowercase "s" in "s3". Uppercase "S", such as remote.S3... is not a valid substitute and causes any attempted update to the setting's value to not take effect.

Compress tsidx files upon upload to S3

To reduce S3 usage and improve network performance, SmartStore can compress tsidx files before uploading them to the remote store. This capability uses zstd compression. When the files are subsequently downloaded to indexers, SmartStore will automatically decompress the files before placing them in the cache.

To enable tsidx file compression, set the remote.s3.tsidx_compression setting in indexes.conf to true. The setting default is false.

This capability is available for AWS S3 and S3-compliant remote stores only.

This feature is not backward compatible. If you downgrade your indexers to a pre-9.0 version after enabling tsidx compression, the indexers will be unable to use the compressed tsidx files.

If enabling this setting on an indexer cluster, you must make the change on all peer nodes through the configuration bundle method, as for any indexes.conf setting on an indexer cluster. A restart is not required.

Non-SmartStore-specific settings in indexes.conf

You must specify these configurations for all SmartStore indexes:

• repFactor = auto, for indexes on indexer cluster peer nodes (not standalone indexers)
• maxDataSize = auto. This is the default value (750MB), so you usually do not need to set it explicitly.
• homePath requires a path value. This is the location on local storage where hot and cached warm buckets reside.
• coldPath requires a path value, even though the setting is ignored except in the case of migrated indexes.
• thawedPath requires a path value, even though the setting has no practical purpose because you cannot thaw data to a SmartStore index. See Thawing data and SmartStore.

Settings in indexes.conf that are incompatible with SmartStore or otherwise restricted

SmartStore indexers and indexer clusters support a subset of features available with non-SmartStore indexers and indexer clusters. Settings related to incompatible features require special consideration.

In addition, SmartStore indexers and indexer clusters handle certain needs, such as data retention, differently from non-SmartStore indexers and indexer clusters. The associated settings are also different.

For information on features that are unsupported or restricted when using SmartStore, see:

• Features not supported by SmartStore
• Current restrictions on SmartStore use

The following indexes.conf settings must retain their default values:

• createBloomfilter. Do not change from default of true.
• enableOnlineBucketRepair. Do not change from default of true.
• isReadOnly. Do not change from default of false.
• enableTsidxReduction. Do not change from default of false.
• maxDataSize. Do not change from default of auto (recommended).
• bloomHomePath. Do not change the path location.
• summaryHomePath. Do not change the path location.
• tstatsHomePath . Do not change the path location.

The following indexes.conf settings are ignored by SmartStore:

• maxWarmDBCount
• maxTotalDataSizeMB
• warmToColdScript
• homePath.maxDataSizeMB
• coldPath.maxDataSizeMB
• maxVolumeDataSizeMB

The journalCompression setting can be set to either zstd (the default) or gzip.

Path settings and SmartStore

SmartStore uses the settings remotePath and path to identify the location of index storage on the remote store.

All other path settings identify locations on the local cache, not the remote store. These settings include:

• homePath
• coldPath
• thawedPath
• bloomHomePath
• summaryHomePath
• tstatsHomePath

See other sections of this topic for configuration and usage notes for each of these settings.

SmartStore settings in server.conf

The SmartStore-related settings in server.conf control the behavior of indexers, including the functionality of the cache manager. The table lists the most frequently configured settings.

SmartStore server.conf setting	Stanza level	Description
eviction_policy = <string>	[cachemanager]	Sets the policy that determines which buckets the cache manager evicts next. Defaults to "lru". See Configure the SmartStore cache manager.
max_cache_size = <integer>	[cachemanager]	Specifies the maximum space, in megabytes, that the cache can occupy on a disk partition. See Configure the SmartStore cache manager.
eviction_padding = <integer>	[cachemanager]	Specifies the additional space, in megabytes, beyond minFreeSpace that the cache manager uses as the threshold to start evicting data. See Configure the SmartStore cache manager.
hotlist_recency_secs = <integer>	[cachemanager]	Specifies the time period, based on the bucket's age, that the cache manager attempts to protect a recent bucket from eviction. This setting operates on a global level, across all indexes, while the version of the setting in indexes.conf operates on a per-index level. See Configure the SmartStore cache manager.
hotlist_bloom_filter_recency_hours = <integer>	[cachemanager]	Specifies the time period, based on the bucket's age, that the cache manager attempts to protect the bucket's non-journal and non-tsidx files, such as the bloomfilter file, from eviction. This setting operates on a global level, across all indexes, while the version of the setting in indexes.conf operates on a per-index level. See Configure the SmartStore cache manager.
various settings	[clustering]	Includes various, mostly low-level, settings that control remote bucket operations in indexer clusters.
cleanRemoteStorageByDefault = <bool>	[general]	Causes the splunk clean eventdata command to clean the remote indexes. Defaults to "false".

For details on these and other server.conf settings, see server.conf.spec.

Most of the SmartStore server.conf settings, including all of the common ones, are configured on the peer nodes. A few rarely-changed settings are configured instead on the manager node. All settings configured on the peer nodes must be the same across all peer nodes.

SmartStore settings in limits.conf

The limits.conf file contains several low-level search-related settings; for example, bucket_localize_max_timeout_sec. These settings are primarily related to the process of localizing buckets. Do not change the settings without a clear understanding of the underlying processes.

For details on limits.conf settings, see limits.conf.spec.