Splunk® Enterprise

Managing Indexers and Clusters of Indexers

Download manual as PDF

Splunk Enterprise version 6.x is no longer supported as of October 23, 2019. See the Splunk Software Support Policy for details. For information about upgrading to a supported version, see How to upgrade Splunk Enterprise.
This documentation does not apply to the most recent version of Splunk. Click here for the latest version.
Download topic as PDF

View the master dashboard

This dashboard provides detailed information on the status of the entire cluster. You can also get information on each of the master's peer nodes from here.

For information on the other clustering dashboards, read:

Access the dashboard

To view the dashboard for the master node:

1. Click Settings in the upper right corner of Splunk Web on the master.

2. In the Distributed environment group, click Clustering.

You can only view this dashboard on an instance that has already been enabled as a master.

View the dashboard

The master dashboard contains these sections:

Here's how the dashboard initially appears, with the Peers tab open:

Master dashboard bubbles.png

Cluster overview

The Cluster Overview summarizes the health of your cluster. It tells you:

  • whether the cluster's data is fully searchable; that is, whether all buckets in the cluster have a primary copy.
  • whether the search and replication factors have been met.
  • how many peers are searchable.
  • how many indexes are searchable.

Depending on the health of your cluster, it might also provide warning messages such as:

  • Some data is not searchable.
  • Replication factor not met.
  • Search factor not met.

For details on the information presented in the Cluster Overview, look through the tabs below the overview.

On the upper right side of the dashboard, there are three buttons:

  • Edit. For information on this button, see "Perform edits on the master".
  • More Info. This button provides details on the master node configuration:
    • Name. The master's serverName, as specified in the master's $SPLUNK_HOME/etc/system/local/server.conf file.
    • Replication Factor. The cluster's replication factor.
    • Search Factor. The cluster's search factor.
    • Generation ID. The cluster's current generation ID.
  • Documentation.

Peers tab

For each peer, the master dashboard lists:

  • Peer Name. The peer's serverName, as specified in the peer's $SPLUNK_HOME/etc/system/local/server.conf file.
  • Searchable. This column indicates whether the peer is currently searchable.
  • Status. The peer's status. For more information about the processes discussed here, read "Take a peer offline". Possible values include:
    • Up
    • Pending. This occurs when a replication fails. It transitions back to Up on the next successful heartbeat from the peer to the master.
    • Detention. A peer goes into detention when it hits resource constraints (for example, it runs out of disk space). While in detention, a peer does not accept inputs and does not participate in searches.
    • Restarting. When you run the offline command, the peer enters this state temporarily after it leaves the ReassigningPrimaries state. It remains in this state for the restart_timeout period (60 seconds by default). If you do not restart the peer within this time, it then moves to the Down state. The peer also enters this state during rolling restarts or if restarted via Splunk Web.
    • ShuttingDown. The master detects that the peer is shutting down.
    • ReassigningPrimaries. A peer enters this state temporarily when you run the offline command.
    • Down. The peer enters this state when it goes offline: either you ran the version of the offline command and the peer shut down for longer than the restart_timeout period (60 seconds by default), or the peer went offline for some other reason (for instance, it crashed).
  • Buckets. The number of buckets for which the peer has copies.

To get more information for any peer, click on the arrow to the left of the peer name. These fields appear:

  • Location. The peer's IP address and port number.
  • Last Heartbeat. The time of the last heartbeat the master received from the peer.
  • Replication port. The port on which the peer receives replicated data from other peers.
  • Generation ID. The peer's base generation ID, which is equivalent to the cluster's generation ID at the moment that the peer last joined the cluster. This ID will be less or equal to the cluster's current generation ID. So, if a peer joined the cluster at generation 1 and has stayed in the cluster ever since, its base generation ID remains 1, even though the cluster might have incremented its current generation ID to, say, 5.
  • GUID. The peer's GUID.

Indexes tab

For each index, the master dashboard lists:

  • Index Name. The name of the index. Internal indexes are preceded by an underscore (_).
  • Searchable. Is the index searchable? In other words, does it have at least one searchable copy of each bucket? If even one bucket in the index does not have a searchable copy, this field will report the index as non-searchable.
  • Searchable Data Copies. The number of complete searchable copies of the index that the cluster has.
  • Replicated Data Copies. The number of copies of the index that the cluster has. Each copy must be complete, with no buckets missing.
  • Buckets. The number of buckets in the index.
  • Cumulative Raw Data Size. The size of the index, excluding hot buckets.

The list of indexes include the internal indexes, _audit and _internal. As you would expect in a cluster, these internal indexes contain the combined data generated by all peers in the cluster. If you need to search for the data generated by a single peer, you can search on the peer's host name.

Search heads tab

For each search head accessing this cluster, the master dashboard lists:

  • Search Head Name. The search head's serverName, as specified in the peer's $SPLUNK_HOME/etc/system/local/server.conf file.
  • Status. Is the search head up?

Note: The list includes the master node as one of the search heads. While the master has search head capabilities, you should only use those capabilities for debugging purposes. The resources of the master must be dedicated to fulfilling its critical role of coordinating cluster activities. Under no circumstances should the master be employed as a production search head. Also, unlike a dedicated search head, the search head on the master cannot be configured for multi-cluster search; it can only search its own cluster.

To get more information for any search head, click on the arrow to the left of the search head name. These fields appear:

  • Location. The search head's server name and port number.
  • GUID. The search head's GUID.

Perform edits on the master

The Edit button at the top right of the master dashboard offers several options:

  • Node Type. Change the instance's node type. Warning: It is extremely unlikely that you will want to change the node type for nodes in an active cluster. Consider the consequences carefully before doing so.
  • Master Node Configuration. Change these master node settings:
    • Replication Factor. Change the cluster's replication factor. Warning: It is inadvisable to increase the replication factor once your cluster contains significant amounts of data. Doing so will kick off a great deal of bucket activity, which will have an adverse effect on the cluster's performance while bucket copies are being created.
    • Search Factor. Change the cluster's search factor. Warning: It is inadvisable to increase the replication factor once your cluster contains significant amounts of data. Doing so will kick off a great deal of bucket activity, which will have an adverse effect on the cluster's performance while bucket copies are being made searchable.
    • Security Key. Change the security key. Only change the secret key if you are also changing it for all other nodes in the cluster. The key must be the same across all instances in a cluster.
  • Distribute Configuration Bundle. Distribute updated configurations and apps to the set of peer nodes. For details of this process, see the topic "Update common peer configurations and apps".
  • Disable Clustering. Remove this node from the cluster. Warning: If you remove the master node from the cluster, the entire cluster will eventually fail.
Last modified on 28 June, 2016
Configure the cluster with the CLI
View the peer dashboard

This documentation applies to the following versions of Splunk® Enterprise: 6.0, 6.0.1, 6.0.2, 6.0.3, 6.0.4, 6.0.5, 6.0.6, 6.0.7, 6.0.8, 6.0.9, 6.0.10, 6.0.11, 6.0.12, 6.0.13, 6.0.14, 6.0.15

Was this documentation topic helpful?

Enter your email address, and someone from the documentation team will respond to you:

Please provide your comments here. Ask a question or make a suggestion.

You must be logged into splunk.com in order to post comments. Log in now.

Please try to keep this discussion focused on the content covered in this documentation topic. If you have a more general question about Splunk functionality or are experiencing a difficulty with Splunk, consider posting a question to Splunkbase Answers.

0 out of 1000 Characters