Splunk® Enterprise

Managing Indexers and Clusters of Indexers

Download manual as PDF

Splunk Enterprise version 5.0 reached its End of Life on December 1, 2017. Please see the migration information.
This documentation does not apply to the most recent version of Splunk. Click here for the latest version.
Download topic as PDF

Use multiple partitions for index data

Splunk can use multiple disks and partitions for its index data. It's possible to configure Splunk to use many disks/partitions/filesystems on the basis of multiple indexes and bucket types, so long as you mount them correctly and point to them properly from indexes.conf. However, we recommend that you use a single high performance file system to hold your Splunk index data for the best experience.

If you do use multiple partitions, the most common way to arrange Splunk's index data is to keep the hot/warm buckets on the local machine, and to put the cold bucket on a separate array of disks (for longer term storage). You'll want to run your hot/warm buckets on a machine with with fast read/write partitions, since most searching will happen there. Cold buckets should be located on a reliable array of disks.

Important: Requirements for cold storage are entirely different if you're using clusters, as described below in "Clusters and the coldPath".

Configure multiple partitions

To configure multiple partitions:

1. Set up partitions just as you'd normally set them up in any operating system.

2. Mount the disks/partitions.

3. Edit indexes.conf to point to the correct paths for the partitions. You set paths on a per-index basis, so you can also set separate partitions for different indexes. Each index has its own [<index>] stanza, where <index> is the name of the index. These are the settable path attributes:

  • homePath = <path on server>
    • This is the path that contains the hot and warm databases for the index.
    • Caution: The path must be writable.
  • coldPath = <path on server>
    • This is the path that contains the cold databases for the index.
    • Caution: The path must be writable.
    • Important: In a cluster, this path also serves as the location of all replicated copies of buckets - hot, warm, and cold. (The original copies of cluster buckets, however, reside in their normal locations, according to the type of bucket.) Therefore, the type of storage you use for the coldPath directory has entirely different requirements with clusters, as described below, in "Clusters and the coldPath".
  • thawedPath = <path on server>
    • This is the path that contains any thawed databases for the index.

Clusters and the coldPath

In a cluster, the storage used for the coldPath directory should have the same characteristics as that used for homePath storage. This is because all replicated copies of buckets reside in the coldPath directory. It doesn't matter whether they're hot, warm, or cold. If you use slower storage for the coldPath location, it will slow the overall performance of your cluster.

Unlike non-clustered indexers, where coldPath typically contains infrequently accessed data and can therefore be located on slower disk arrays, clusters require strongly performing storage for the coldPath location, to handle the needs of cluster operations. For example, some of the buckets in the coldPath location on a cluster peer will be replicated hot bucket copies still being written to. Other buckets will be replicated warm copies, and the search head might be accessing them frequently. In addition, depending on how the cluster is configured and what occurs subsequently (in terms of peers going offline, etc.), the peer might need to convert bucket copies from non-searchable to searchable, entailing a considerable amount of processing on the coldPath data.

For more information on cluster operations, read "About clusters and index replication" and the topics that follow it. In particular, the topic "System requirements" has detailed information about cluster storage hardware.

Move the index database
Configure maximum index size

This documentation applies to the following versions of Splunk® Enterprise: 5.0, 5.0.1, 5.0.2, 5.0.3, 5.0.4, 5.0.5, 5.0.6, 5.0.7, 5.0.8, 5.0.9, 5.0.10, 5.0.11, 5.0.12, 5.0.13, 5.0.14, 5.0.15, 5.0.16, 5.0.17, 5.0.18

Was this documentation topic helpful?

Enter your email address, and someone from the documentation team will respond to you:

Please provide your comments here. Ask a question or make a suggestion.

You must be logged into splunk.com in order to post comments. Log in now.

Please try to keep this discussion focused on the content covered in this documentation topic. If you have a more general question about Splunk functionality or are experiencing a difficulty with Splunk, consider posting a question to Splunkbase Answers.

0 out of 1000 Characters