Splunk® Enterprise

Managing Indexers and Clusters of Indexers

Download manual as PDF

Splunk Enterprise version 5.0 reached its End of Life on December 1, 2017. Please see the migration information.
This documentation does not apply to the most recent version of Splunk. Click here for the latest version.
Download topic as PDF

Configure the peer indexes

You configure indexes by editing the indexes.conf file. This file determines an indexer's set of indexes, as well as the size and attributes of its buckets. Since all peers in a cluster must use the same set of indexes (except for limited purposes, described later), the indexes.conf file should ordinarily be the same across all peers.

Note: Like most Splunk configuration files, you can have multiple versions of the indexes.conf file, located in several directories on your Splunk instance. For more information on configuration files and how the system layers multiple versions, read "About configuration files".

The cluster peers deploy with a default indexes.conf file that handles basic cluster needs. If you want to add indexes or change bucket behavior, you edit a new indexes.conf file in a special location on the master and then distribute the file simultaneously to all the peers.

Important: You cannot use Manager or the CLI to configure index settings on peer nodes. You must edit indexes.conf directly.

All peers must use the same set of indexes.conf files

The set of indexes.conf files should ordinarily be identical across all peers in a cluster. In particular, all peers must use the same set of clustered indexes. This is essential for index replication to work properly. (The master node, on the other hand, uses its own, separate indexes.conf file, because it indexes only its own internal data.) There's a limited exception to this stricture, which is described a bit later.

When you first create the cluster, the master distributes a default indexes.conf file to each of the peers. This version of indexes.conf is specifically designed for cluster peers. The default indexes.conf turns on replication for the main (default) index, as well as the internal indexes, such as _audit, _internal, etc.

Depending on your system, you might also need to edit and distribute a modified indexes.conf to the peers, to accommodate additional indexes or changes to bucket attributes. To ensure that all peers use the same indexes.conf, you must use the master node to distribute the file to all the peers as a single process.This process, known as the configuration bundle method, is described in "Update common peer configurations".

Although it's possible to distribute the indexes.conf file to all peers using some other distribution method, only the configuration bundle method is recommended and supported. When you use this method, the master orchestrates the distribution to ensure that all peers have the same set of clustered indexes. If you use another distribution method, you must ensure that settings for any new clustered indexes are successfully distributed to all peers and that all the peers have been restarted, before you start sending data to the new indexes.

You can also use the configuration bundle method to distribute apps across all the peers. These apps might contain their own indexes.conf files, which will layer appropriately with any non-app version of the file that you might also distribute to the peers. For information on app distribution, read "How to distribute apps to all the peers".

Note: Under limited circumstances (for example, to perform local testing or monitoring), you can create a peer-specific indexes.conf that configures an index for that peer only. However, such an index will not get replicated. The peer-specific indexes.conf supplements, but does not replace, the common version of the file that all peers get. See "Add an index to a single peer" for details.

Configure a set of indexes for the peers

There are two steps to configuring indexes across the set of peers:

1. Edit a common indexes.conf on the master. See "Edit indexes.conf".

2. Use the master to distribute the file across the set of peers. See "Distribute the new indexes.conf file to the peers".

These two steps are described below.

Edit indexes.conf

For details on configuring indexes.conf, read the topics in the chapters "Manage indexes" and "Manage index storage" in this manual. For a list of all indexes.conf attributes, see the indexes.conf specification file in the Admin Manual.

For the most part, you edit the cluster peer indexes.conf the same as for any Splunk indexer. However, there are a few differences to be aware of.

The indexes.conf repFactor attribute

When you add a new index stanza, you must set the repFactor attribute to auto. This causes that the index's data to be replicated to other peers in the cluster. For example:

homePath=<path for hot and warm buckets>
coldPath=<path for cold buckets>
thawedPath=<path for thawed buckets>

Specify homePath and coldPath with forward-slash directory separators

In heterogeneous environments, it's possible that the master node's operating system could use a different convention for specifying directory paths from the peer nodes' operating system. This presents a problem because you edit the indexes.conf file on the master but then you distribute it to the peers.

For example, if you have a Windows master and a set of Linux peers, the normal way to specify the homePath on the Windows master, where the file gets edited, would be to use the Windows backward-slash convention as a directory separator, while the Linux peers, where the file gets distributed, require forward slashes.

To deal with this possibility, the best practice is to always use forward slashes when specifying directory paths in homePath and coldPath, no matter which operating systems your master and peers use. For example:

homePath = $SPLUNK_HOME/var/lib/splunk/defaultdb/db/

Splunk will always accept the forward slash as a directory separator.

Use index volumes to simplify storage management

Important: To understand this subtopic, you need to be familiar with the specifics of cluster storage requirements, as described in "Storage considerations".

Index volumes provide a handy way to designate storage size across multiple indexes. Like all other index settings, you specify volumes in indexes.conf. See "Configure index size with volumes" for information on index volumes and how to specify them.

In addition to the use cases presented in that section, you can also use volumes to designate the total size of combined hot/warm and cold data across the entire set of indexes. This is particularly useful for clusters, where replicated data resides in cold databases and it can be difficult to predict the mix of original and replicated data that will exist on any one peer node. By specifying a single volume for the combined sets of hot/warm and cold databases, you can control the overall amount of storage that your indexes occupy on each peer, even though you cannot safely control the amount that's dedicated to a specific bucket type.

This example uses a volume to designate the combined storage size of all buckets (original and replicated) for two indexes:

path = /mnt/big_disk
maxVolumeDataSizeMB = 300000

homePath = volume:cluster1/idx1/db
coldPath = volume:cluster1/idx1/colddb
thawedPath=<path for thawed buckets>

homePath = volume:cluster1/idx2/db
coldPath = volume:cluster1/idx2/colddb
thawedPath=<path for thawed buckets>

Since this configuration is part of indexes.conf and so will be identical on all peers in the cluster, you must designate the volume's path attribute in a way that works across all the peers. (It's not that all peers should use the same physical storage - in fact, they should not - but, rather, the path reference that you specify must resolve properly on each peer.) In addition, you cannot use volumes to specify the thawed path.

Note: While this approach works well for clustered storage, since the hot/warm and cold databases in clusters must have similar performance characteristics, it is not a recommended approach for non-clustered indexers, where you typically want to specify slower storage for the cold database. See "Configure index size with volumes" for details.

Distribute the new indexes.conf file to the peers

Once you've edited indexes.conf, you need to distribute it to the cluster's set of peer nodes. To learn how to distribute configuration files, including indexes.conf, across all the peers, read "Update common peer configurations".

For information about other types of peer configuration, including app distribution, read "Configure the peer nodes".

Configure the peer nodes
Configure the search head

This documentation applies to the following versions of Splunk® Enterprise: 5.0.2, 5.0.3, 5.0.4, 5.0.5, 5.0.6, 5.0.7, 5.0.8, 5.0.9, 5.0.10, 5.0.11, 5.0.12, 5.0.13, 5.0.14, 5.0.15, 5.0.16, 5.0.17, 5.0.18

Was this documentation topic helpful?

Enter your email address, and someone from the documentation team will respond to you:

Please provide your comments here. Ask a question or make a suggestion.

You must be logged into splunk.com in order to post comments. Log in now.

Please try to keep this discussion focused on the content covered in this documentation topic. If you have a more general question about Splunk functionality or are experiencing a difficulty with Splunk, consider posting a question to Splunkbase Answers.

0 out of 1000 Characters