Splunk® Enterprise

Managing Indexers and Clusters of Indexers

Download manual as PDF

This documentation does not apply to the most recent version of Splunk. Click here for the latest version.
Download topic as PDF

Configure index storage

You configure indexes in indexes.conf. How you edit indexes.conf depends on whether you're using index replication, also known as indexer clustering:

  • For non-clustered indexes, edit a copy of indexes.conf in $SPLUNK_HOME/etc/system/local/ or in a custom app directory in $SPLUNK_HOME/etc/apps/. Do not edit the copy in $SPLUNK_HOME/etc/system/default. For information on configuration files and directory locations, see About configuration files.
    You can also use Splunk Web to configure the path to your indexes. Go to Settings > System settings > General settings. Under the section Index settings, set the field Path to indexes. After doing this, you must restart the indexer from the CLI, not from within Splunk Web. This method is only available for non-clustered indexes.

Attributes that affect index buckets

This table lists the key indexes.conf attributes affecting buckets and what they configure. It also provides links to other topics that show how to use these attributes. For the most detailed information on these attributes, as well as others, always refer to the indexes.conf spec file.

Attribute What it configures Default For more information, see ...
homePath The path that contains the hot and warm buckets. (Required.)

This location must be writable.

$SPLUNK_HOME/var/lib/splunk/ defaultdb/db/ (for the default index only) Use multiple partitions for index data
coldPath The path that contains the cold buckets. (Required.)

This location must be writable.

$SPLUNK_HOME/var/lib/splunk/ defaultdb/colddb/ (for the default index only) Use multiple partitions for index data
thawedPath The path that contains any thawed buckets. (Required.)

This location must be writable.

$SPLUNK_HOME/var/lib/splunk/ defaultdb/thaweddb/ (for the default index only) Use multiple partitions for index data
repFactor Determines whether the index gets replicated to other cluster peers. (Required for indexes on cluster peer nodes.) 0 (which means that the index will not get replicated to other peers; the correct behavior for non-clustered indexes). For clustered indexes, you must set repFactor to auto, which causes the index to get replicated. Configure the peer indexes in an indexer cluster
maxHotBuckets The maximum number of hot buckets. This value should be at least 2, to deal with any archival data. The main default index, for example, has this value set to 10. 3, for new, custom indexes. How data ages
maxDataSize Determines rolling behavior, hot to warm. The maximum size for a hot bucket. When a hot bucket reaches this size, it rolls to warm. This attribute also determines the approximate size for all buckets. Depends; see indexes.conf. Use multiple partitions for index data

Set a retirement and archiving policy

maxWarmDBCount Determines rolling behavior, warm to cold. The maximum number of warm buckets. When the maximum is reached, warm buckets begin rolling to cold. 300 Use multiple partitions for index data
maxTotalDataSizeMB Determines rolling behavior, cold to frozen. The maximum size of an index. When this limit is reached, cold buckets begin rolling to frozen. 500000 (MB) Set a retirement and archiving policy
frozenTimePeriodInSecs Determines rolling behavior, cold to frozen. Maximum age for a bucket, after which it rolls to frozen. 188697600 (in seconds; approx. 6 years) Set a retirement and archiving policy
coldToFrozenDir Location for archived data. Determines behavior when a bucket rolls from cold to frozen. If set, the indexer will archive frozen buckets into this directory just before deleting them from the index. If you don't set either this attribute or coldToFrozenScript, the indexer will just log the bucket's directory name and then delete it once it rolls to frozen. Archive indexed data
coldToFrozenScript Script to run just before a cold bucket rolls to frozen. If you set both this attribute and coldToFrozenDir, the indexer will use coldToFrozenDir and ignore this attribute. If you don't set either this attribute or coldToFrozenDir, the indexer will just log the bucket's directory name and then delete it once it rolls to frozen. Archive indexed data
homePath.maxDataSizeMB

coldPath.maxDataSizeMB

Maximum size for homePath (hot/warm bucket storage) or coldPath (cold bucket storage). If either attribute is missing or set to 0, its path is not individually constrained in size. None Configure index size according to bucket type
maxVolumeDataSizeMB Maximum size for a volume. If the attribute is missing, the individual volume is not constrained in size. None Configure index size with volumes

Index size and indexer clusters

The attributes that control the size of an index and the number of buckets operate on each peer node individually. They do not operate across the cluster.

For example, take the maxTotalDataSizeMB attribute. This attribute specifies the maximum size of the index. This value is applied on a per-peer basis. When an index reaches its maximum size on a particular peer node, the peer freezes the oldest bucket in its copy of the index.

This means that the size of an index on a peer node is determined by the total size of all bucket copies for that index on that peer node. It doesn't matter whether the copies are primary copies, searchable copies, non-searchable copies, or excess copies. They all count toward the index size on that peer.

Because you cannot ordinarily control the distribution of replicated copies across the cluster or specify whether a particular copy is searchable, an index will typically be a different size on each of the peer nodes. This means that the index might reach its maximum size on one peer while still having room to grow on the other peers.

To handle this situation, each peer tells the master when it freezes a bucket. The master does not tell the other peers to freeze any copies that they might have of that bucket. However, it tracks the frozen buckets and does not perform fix-up on any buckets that have been frozen on one of the peers. The peers will individually roll any copies they have of that bucket when they subsequently reach the maximum size limit. See How the cluster handles frozen buckets.

Note: Although these attributes operate separately on each peer, you should set them to the same values across all peers in the cluster. See Configure the peer indexes in an indexer cluster.

For help in sizing your cluster disk space needs, see Storage considerations.

PREVIOUS
How the indexer stores indexes
  NEXT
Move the index database

This documentation applies to the following versions of Splunk® Enterprise: 6.0, 6.0.1, 6.0.2, 6.0.3, 6.0.4, 6.0.5, 6.0.6, 6.0.7, 6.0.8, 6.0.9, 6.0.10, 6.0.11, 6.0.12, 6.0.13, 6.0.14, 6.0.15, 6.1, 6.1.1, 6.1.2, 6.1.3, 6.1.4, 6.1.5, 6.1.6, 6.1.7, 6.1.8, 6.1.9, 6.1.10, 6.1.11, 6.1.12, 6.1.13, 6.1.14


Was this documentation topic helpful?

Enter your email address, and someone from the documentation team will respond to you:

Please provide your comments here. Ask a question or make a suggestion.

You must be logged into splunk.com in order to post comments. Log in now.

Please try to keep this discussion focused on the content covered in this documentation topic. If you have a more general question about Splunk functionality or are experiencing a difficulty with Splunk, consider posting a question to Splunkbase Answers.

0 out of 1000 Characters