Configure maximum index size

Note: This topic is not relevant to SmartStore indexes. See Configure data retention for SmartStore indexes.

You can configure maximum index size in a number of ways:

• On a per-index basis
• For hot/warm and cold buckets separately
• Across indexes, using volumes

To configure index storage size, you set attributes in indexes.conf. For more information on the attributes mentioned in this topic, read "Configure index storage".

Caution: While processing indexes, the indexer might occasionally exceed the configured maximums for short periods of time. When setting limits, be sure to factor in some buffer space. Also, note that certain systems, such as most Unix systems, maintain a configurable reserve space on their partitions. You must take that reserve space, if any, into account when determining how large your indexes can grow.

Configure index size for each index

To set the maximum index size on a per-index basis, use the maxTotalDataSizeMB attribute. When this limit is reached, buckets begin rolling to frozen.

Configure index size according to bucket type

To set the maximum size for homePath (hot/warm bucket storage) or coldPath (cold bucket storage), use the maxDataSizeMB settings:

# set hot/warm storage to 10,000MB
homePath.maxDataSizeMB = 10000 
# set cold storage to 5,000MB
coldPath.maxDataSizeMB = 5000

The maxDataSizeMB attributes can be set globally or for each index. An index-level setting will override a global setting. To control bucket storage across groups of indexes, use the maxVolumeDataSizeMB attribute, described below.

When the size of the homePath directory exceeds homePath.maxDataSizeMB, the indexer rolls the oldest warm bucket to cold, moving the bucket to the coldPath directory.

When the size of the coldPath directory exceeds coldPath.maxDataSizeMB, the indexer rolls the oldest cold bucket to frozen.

Configure index size with volumes

You can manage disk usage across multiple indexes by creating volumes and specifying maximum data size for them. A volume represents a directory on the file system where indexed data resides.

Volumes can store data from multiple indexes. You would typically use separate volumes for hot/warm and cold buckets. For instance, you can set up one volume to contain the hot/warm buckets for all your indexes, and another volume to contain the cold buckets.

You can use volumes to define homePath and coldPath. You cannot use them to define thawedPath.

In addition, you must use volumes if you explicitly define bloomHomePath.

Configure a volume

To set up a volume, use this syntax:

[volume:<volume_name>]
path = <pathname_for_volume>

You can also optionally include a maxVolumeDataSizeMB attribute, which specfies the maximum size for the volume.

For example:

[volume:hot1]
path = /mnt/fast_disk
maxVolumeDataSizeMB = 100000

The example defines a volume called "hot1", located at /mnt/fast_disk, with a maximum size of 100,000MB.

Similarly, this stanza defines a volume called "cold1" that uses a maximum of 150,000MB:

[volume:cold1]
path = /mnt/big_disk
maxVolumeDataSizeMB = 150000

Use a volume

Once you configure volumes, you can use them to define an index's homePath and coldPath. For example, using the volumes configured above, you can define two indexes:

[idx1]
homePath = volume:hot1/idx1
coldPath = volume:cold1/idx1

[idx2]
homePath = volume:hot1/idx2
coldPath = volume:cold1/idx2

You can use volumes to manage index storage space in any way that makes sense to you. Usually, however, volumes correlate to hot/warm and cold buckets, because of the different storage requirements typical when dealing with different bucket types. So, you will probably use some volumes exclusively for designating homePath (hot/warm buckets) and others for coldPath (cold buckets).

When a volume containing warm buckets reaches its maxVolumeDataSizeMB, it starts rolling buckets to cold. When a volume containing cold buckets reaches its maxVolumeDataSizeMB, it starts rolling buckets to frozen. If a volume contains both warm and cold buckets (which will happen if an index's homePath and coldPath are both set to the same volume), the oldest bucket will be rolled to frozen.

Put it all together

This example shows how to use the per-index homePath.maxDataSizeMB and coldPath.maxDataSizeMB attributes in combination with volumes to maintain fine-grained control over index storage. In particular, it shows how to use those attributes to prevent bursts of data into one index from triggering massive bucket moves from other indexes. You can use these per-index settings to ensure that no index will ever occupy more than a specified size, thereby alleviating that concern.

# global settings

# Inheritable by all indexes: No hot/warm directory (homePath) can exceed 1 TB.
# Individual indexes can override this setting.
homePath.maxDataSizeMB = 1000000

# volumes

[volume:caliente]
path = /mnt/fast_disk
maxVolumeDataSizeMB = 100000

[volume:frio]
path = /mnt/big_disk
maxVolumeDataSizeMB = 1000000

# indexes

[i1]
homePath = volume:caliente/i1
# homePath.maxDataSizeMB is inherited from the global setting
coldPath = volume:frio/i1
# coldPath.maxDataSizeMB not specified anywhere: 
# This results in no size limit - old-style behavior

[i2]
homePath = volume:caliente/i2
homePath.maxDataSizeMB = 1000  
# overrides the global default
coldPath = volume:frio/i2
coldPath.maxDataSizeMB = 10000  
# limits the size of cold buckets

[i3]
homePath = /old/style/path
homePath.maxDataSizeMB = 1000
coldPath = volume:frio/i3
coldPath.maxDataSizeMB = 10000