Configure maximum index size
Note: This topic is not relevant to SmartStore indexes. See Configure data retention for SmartStore indexes.
You can configure maximum index size in a number of ways:
- On a per-index basis
- For hot/warm and cold buckets separately
- Across indexes, using volumes
To configure index storage size, you set attributes in indexes.conf. For more information on the attributes mentioned in this topic, read "Configure index storage".
Caution: While processing indexes, the indexer might occasionally exceed the configured maximums for short periods of time. When setting limits, be sure to factor in some buffer space. Also, note that certain systems, such as most Unix systems, maintain a configurable reserve space on their partitions. You must take that reserve space, if any, into account when determining how large your indexes can grow.
Configure index size for each index
To set the maximum index size on a per-index basis, use the
maxTotalDataSizeMB attribute. When this limit is reached, buckets begin rolling to frozen.
Configure index size according to bucket type
To set the maximum size for
homePath (hot/warm bucket storage) or
coldPath (cold bucket storage), use the
# set hot/warm storage to 10,000MB homePath.maxDataSizeMB = 10000 # set cold storage to 5,000MB coldPath.maxDataSizeMB = 5000
maxDataSizeMB attributes can be set globally or for each index. An index-level setting will override a global setting. To control bucket storage across groups of indexes, use the
maxVolumeDataSizeMB attribute, described below.
When the size of the
homePath directory exceeds
homePath.maxDataSizeMB, the indexer rolls the oldest warm bucket to cold, moving the bucket to the
When the size of the
coldPath directory exceeds
coldPath.maxDataSizeMB, the indexer rolls the oldest cold bucket to frozen.
Configure index size with volumes
You can manage disk usage across multiple indexes by creating volumes and specifying maximum data size for them. A volume represents a directory on the file system where indexed data resides.
Volumes can store data from multiple indexes. You would typically use separate volumes for hot/warm and cold buckets. For instance, you can set up one volume to contain the hot/warm buckets for all your indexes, and another volume to contain the cold buckets.
You can use volumes to define
coldPath. You cannot use them to define
In addition, you must use volumes if you explicitly define
Configure a volume
To set up a volume, use this syntax:
[volume:<volume_name>] path = <pathname_for_volume>
You can also optionally include a
maxVolumeDataSizeMB attribute, which specfies the maximum size for the volume.
[volume:hot1] path = /mnt/fast_disk maxVolumeDataSizeMB = 100000
The example defines a volume called "hot1", located at
/mnt/fast_disk, with a maximum size of 100,000MB.
Similarly, this stanza defines a volume called "cold1" that uses a maximum of 150,000MB:
[volume:cold1] path = /mnt/big_disk maxVolumeDataSizeMB = 150000
Use a volume
Once you configure volumes, you can use them to define an index's
coldPath. For example, using the volumes configured above, you can define two indexes:
[idx1] homePath = volume:hot1/idx1 coldPath = volume:cold1/idx1 [idx2] homePath = volume:hot1/idx2 coldPath = volume:cold1/idx2
You can use volumes to manage index storage space in any way that makes sense to you. Usually, however, volumes correlate to hot/warm and cold buckets, because of the different storage requirements typical when dealing with different bucket types. So, you will probably use some volumes exclusively for designating
homePath (hot/warm buckets) and others for
coldPath (cold buckets).
When a volume containing warm buckets reaches its
maxVolumeDataSizeMB, it starts rolling buckets to cold. When a volume containing cold buckets reaches its
maxVolumeDataSizeMB, it starts rolling buckets to frozen. If a volume contains both warm and cold buckets (which will happen if an index's
coldPath are both set to the same volume), the oldest bucket will be rolled to frozen.
Put it all together
This example shows how to use the per-index
coldPath.maxDataSizeMB attributes in combination with volumes to maintain fine-grained control over index storage. In particular, it shows how to use those attributes to prevent bursts of data into one index from triggering massive bucket moves from other indexes. You can use these per-index settings to ensure that no index will ever occupy more than a specified size, thereby alleviating that concern.
# global settings # Inheritable by all indexes: No hot/warm directory (homePath) can exceed 1 TB. # Individual indexes can override this setting. homePath.maxDataSizeMB = 1000000 # volumes [volume:caliente] path = /mnt/fast_disk maxVolumeDataSizeMB = 100000 [volume:frio] path = /mnt/big_disk maxVolumeDataSizeMB = 1000000 # indexes [i1] homePath = volume:caliente/i1 # homePath.maxDataSizeMB is inherited from the global setting coldPath = volume:frio/i1 # coldPath.maxDataSizeMB not specified anywhere: # This results in no size limit - old-style behavior [i2] homePath = volume:caliente/i2 homePath.maxDataSizeMB = 1000 # overrides the global default coldPath = volume:frio/i2 coldPath.maxDataSizeMB = 10000 # limits the size of cold buckets [i3] homePath = /old/style/path homePath.maxDataSizeMB = 1000 coldPath = volume:frio/i3 coldPath.maxDataSizeMB = 10000
Use multiple partitions for index data
Set limits on disk usage
This documentation applies to the following versions of Splunk® Enterprise: 7.2.0, 7.2.1, 7.2.2, 7.2.3, 7.2.4, 7.2.5, 7.2.6, 7.2.7, 7.2.8, 7.2.9, 7.2.10, 7.3.0, 7.3.1, 7.3.2, 7.3.3, 7.3.4, 7.3.5, 7.3.6, 7.3.7, 7.3.8, 7.3.9, 8.0.0, 8.0.1, 8.0.2, 8.0.3, 8.0.4, 8.0.5, 8.0.6, 8.0.7, 8.0.8, 8.0.9, 8.0.10, 8.1.0, 8.1.1, 8.1.2, 8.1.3, 8.1.4, 8.1.5, 8.1.6, 8.1.7, 8.1.8, 8.1.9, 8.1.10, 8.1.11, 8.1.12, 8.1.13, 8.1.14, 8.2.0, 8.2.1, 8.2.2, 8.2.3, 8.2.4, 8.2.5, 8.2.6, 8.2.7, 8.2.8, 8.2.9, 8.2.10, 8.2.11, 9.0.0, 9.0.1, 9.0.2, 9.0.3, 9.0.4, 9.0.5
Feedback submitted, thanks!