Configure index storage

You configure indexes in indexes.conf. How you edit indexes.conf depends on whether you're using index replication, also known as indexer clustering:

• For non-clustered indexes, edit the version of indexes.conf in $SPLUNK_HOME/etc/system/local/, or create one if it does not already exist there. Do not edit the copy in $SPLUNK_HOME/etc/system/default. For information on configuration files and directory locations, see About configuration files.
• For clustered indexes, create or edit a version of indexes.conf on the cluster manager node and then distribute it to all the peer nodes, as described in Configure the peer indexes in an indexer cluster.

For non-clustered indexes only, you can optionally use Splunk Web to configure the path to your indexes. Go to Settings > Server settings > General settings. Under the section Index settings, set the field Path to indexes. After doing this, you must restart the indexer from the CLI, not from within Splunk Web. Most other settings, however, require direct editing of indexes.conf.

Attributes that affect index buckets

This table lists the key indexes.conf attributes affecting buckets and what they configure. It also provides links to other topics that show how to use these attributes. For the most detailed information on these attributes, as well as others, always refer to the indexes.conf spec file.

Be careful when adjusting settings, such as maxWarmDBCount, so as to cause the reduction in the number of existing warm buckets. The reduction process, if involving a significant number of buckets, has the potential to overwhelm your network. You can minimize the effect of warm bucket reduction by reducing the setting's value gradually.

Note: This list is specific to non-SmartStore indexes. The set of attributes that control SmartStore indexes is significantly different. See Configure SmartStore.

Attribute	What it configures	Default	For more information, see ...
homePath	The path that contains the hot and warm buckets. (Required.) This location must be writable.	$SPLUNK_HOME/var/lib/splunk/ defaultdb/db/ (for the default index only)	Configure index path attributes
coldPath	The path that contains the cold buckets. (Required.) This location must be writable.	$SPLUNK_HOME/var/lib/splunk/ defaultdb/colddb/ (for the default index only)	Configure index path attributes
thawedPath	The path that contains any thawed buckets. (Required.) This location must be writable.	$SPLUNK_HOME/var/lib/splunk/ defaultdb/thaweddb/ (for the default index only)	Configure index path attributes
repFactor	Determines whether the index gets replicated to other cluster peers. (Required for indexes on cluster peer nodes.)	0 (which means that the index will not get replicated to other peers; the correct behavior for non-clustered indexes). For clustered indexes, you must set repFactor to auto, which causes the index to get replicated.	Configure the peer indexes in an indexer cluster
maxHotBuckets	The maximum number of concurrent hot buckets. This value should be at least 2, to deal with any archival data. The main default index, for example, has this value set to 10.	3, for new, custom indexes.	How data ages
maxDataSize	Determines rolling behavior, hot to warm. The maximum size for a hot bucket. When a hot bucket reaches this size, it rolls to warm. This attribute also determines the approximate size for all buckets.	Special value "auto", which sets the size to 750MB.	How data ages
maxWarmDBCount	Determines rolling behavior, warm to cold. The maximum number of warm buckets. When the maximum is reached, warm buckets begin rolling to cold.	300	Use multiple partitions for index data
maxTotalDataSizeMB	Determines rolling behavior, cold to frozen. The maximum size of an index. When this limit is reached, cold buckets begin rolling to frozen.	500000 (MB)	Set a retirement and archiving policy
frozenTimePeriodInSecs	Determines rolling behavior, cold to frozen. Maximum age for a bucket, after which it rolls to frozen.	188697600 (in seconds; approx. 6 years)	Set a retirement and archiving policy
coldToFrozenDir	Location for archived data. Determines behavior when a bucket rolls from cold to frozen. If set, the indexer will archive frozen buckets into this directory just before deleting them from the index.	If you don't set either this attribute or coldToFrozenScript, the indexer will just log the bucket's directory name and then delete it once it rolls to frozen.	Archive indexed data
coldToFrozenScript	Script to run just before a cold bucket rolls to frozen. If you set both this attribute and coldToFrozenDir, the indexer will use coldToFrozenDir and ignore this attribute.	If you don't set either this attribute or coldToFrozenDir, the indexer will just log the bucket's directory name and then delete it once it rolls to frozen.	Archive indexed data
homePath.maxDataSizeMB coldPath.maxDataSizeMB	Maximum size for homePath (hot/warm bucket storage) or coldPath (cold bucket storage). If either attribute is missing or set to 0, its path is not individually constrained in size.	None	Configure index size according to bucket type
maxVolumeDataSizeMB	Maximum size for a volume. If the attribute is missing, the individual volume is not constrained in size.	None	Configure index size with volumes

Configure index path attributes

When creating a new index, you configure several index path attributes, for example, homePath and coldPath. When you configure path attributes, follow these restrictions and recommendations:

• The path must be writable. In the case of homePath, the parent path must also be writable.
• Do not use environment variables in index paths. The only exception to this is SPLUNK_DB.
• The path cannot be a root directory, such as homePath=/myindex or homePath=C:\myindex.
• It is recommended that you specify the path using $_index_name as placeholder for the index name. For example:
  
  

homePath = $SPLUNK_DB/$_index_name/db

At run time, the indexer expands $_index_name to the name of the index. For example, if the index name is "newindex", homePath becomes $SPLUNK_DB/newindex/db.

The set of index path attributes includes:

• homePath
• coldPath
• thawedPath
• bloomHomePath
• summaryHomePath
• tstatsHomePath

For more information on path attributes, see the indexes.conf spec file.

For information on using multiple partitions to hold your index data, see Use multiple partitions for index data .

Index size and indexer clusters

Note: This section pertains to non-SmartStore indexes only. Clusters handle sizing of SmartStore indexes differently. See Configure data retention for SmartStore indexes.

The attributes that control the size of a non-SmartStore index and its number of buckets operate on each peer node individually. They do not operate across the cluster.

For example, consider the maxTotalDataSizeMB attribute. This attribute specifies the maximum size of the index. Its value is applied on a per-peer basis to limit the size of the index on each peer. When an index reaches its maximum size on a particular peer node, the peer freezes the oldest bucket in its copy of the index.

This means that the size of an index on a peer node is determined by the total size of all bucket copies for that index on that peer node. It doesn't matter whether the copies are primary copies, searchable copies, non-searchable copies, or excess copies. They all count toward the index size on that peer.

Because a cluster usually does not distribute bucket copies perfectly evenly across the set of peer nodes, an index typically has a different size on each peer node. This means that the index might reach its maximum size on one peer while still having room to grow on the other peers.

To handle this situation, each peer tells the manager when it freezes a copy of a bucket. At that point, the manager no longer initiates fix-up activities for the frozen bucket. The manager does not, however, instruct the other peers to freeze their copies of that bucket. Each peer will subsequently freeze its copy of the bucket, if any, when its copy of the index reaches the maximum size limit. See How the cluster handles frozen buckets.

Note: Although these attributes operate separately on each peer, you should set them to the same values across all peers in the cluster. See Configure the peer indexes in an indexer cluster.

For help in sizing your cluster disk space needs, see Storage considerations.

Rawdata journal compression

When the indexer indexes data, it writes the data to the rawdata journal. The indexer compresses the data during this process. By default, the compression uses the zstd algorithm, but you can change the compression algorithm through the journalCompression setting in indexes.conf. Available compression algorithms are:

• gzip
• lz4
• zstd

If you change the compression algorithm, journals in new buckets will be compressed using the changed method, but journals in existing buckets will continue to be stored through the compression method with which they were originally indexed.