Managing Indexers and Clusters of Indexers

View the cluster master dashboard

This dashboard provides detailed information on the status of the entire cluster. You can also get information on each of the master's peer nodes from here.

For information on the other clustering dashboards, read:

• "View the peer dashboard"
• "View the search head dashboard"

Access the dashboard

To view the dashboard for the master node:

1. Click Settings in the upper right corner of Splunk Web on the master.

2. In the Distributed environment group, click Clustering.

You can only view this dashboard on an instance that has been enabled as a master.

View the dashboard

The master dashboard contains these sections:

• Cluster overview
• Peers tab
• Indexes tab
• Search Heads tab

Here is how the dashboard initially appears, with the Peers tab open:

Cluster overview

The cluster overview summarizes the health of your cluster. It tells you:

• whether the cluster's data is fully searchable; that is, whether all buckets in the cluster have a primary copy.
• whether the search and replication factors have been met.
• how many peers are searchable.
• how many indexes are searchable.

Depending on the health of your cluster, it might also provide warning messages such as:

• Some data is not searchable.
• Replication factor not met.
• Search factor not met.

For details on the information presented in the cluster overview, browse the tabs underneath.

On the upper right side of the dashboard, there are three buttons:

• Edit. For information on this button, see "Configure the master with the dashboard.". This button is disabled for multisite clusters.

• More Info. This button provides details on the master node configuration:
  • Name. The master's serverName, as specified in the master's $SPLUNK_HOME/etc/system/local/server.conf file.
  • Replication Factor. The cluster's replication factor.
  • Search Factor. The cluster's search factor.
  • Generation ID. The cluster's current generation ID.
• Documentation.

Peers tab

For each peer, the master dashboard lists:

• Peer Name. The peer's serverName, as specified in the peer's $SPLUNK_HOME/etc/system/local/server.conf file.
• Fully searchable. This column indicates whether the peer currently has a complete set of primaries and is fully searchable.
• Site. (For multisite only.) This column displays the site value for each peer.
• Status. The peer's status. For more information about the processes discussed here, read "Take a peer offline". Possible values include:
  • Up
  • Pending. This occurs when a replication fails. It transitions back to Up on the next successful heartbeat from the peer to the master.
  • Detention. A peer goes into detention when it hits resource constraints (for example, it runs out of disk space). While in detention, a peer does not accept inputs and does not participate in searches.
  • Restarting. When you run the offline command, the peer enters this state temporarily after it leaves the ReassigningPrimaries state. It remains in this state for the restart_timeout period (60 seconds by default). If you do not restart the peer within this time, it then moves to the Down state. The peer also enters this state during rolling restarts or if restarted via Splunk Web.
  • ShuttingDown. The master detects that the peer is shutting down.
  • ReassigningPrimaries. A peer enters this state temporarily when you run the offline command.
  • Down. The peer enters this state when it goes offline: either you ran the version of the offline command and the peer shut down for longer than the restart_timeout period (60 seconds by default), or the peer went offline for some other reason (for instance, it crashed).
• Buckets. The number of buckets for which the peer has copies.

To get more information for any peer, click on the arrow to the left of the peer name. These fields appear:

• Location. The peer's IP address and port number.
• Last Heartbeat. The time of the last heartbeat the master received from the peer.
• Replication port. The port on which the peer receives replicated data from other peers.
• Base generation ID. The peer's base generation ID, which is equivalent to the cluster's generation ID at the moment that the peer last joined the cluster. This ID will be less or equal to the cluster's current generation ID. So, if a peer joined the cluster at generation 1 and has stayed in the cluster ever since, its base generation ID remains 1, even though the cluster might have incremented its current generation ID to, say, 5.
• GUID. The peer's GUID.

Indexes tab

For each index, the master dashboard lists:

• Index Name. The name of the index. Internal indexes are preceded by an underscore (_).
• Fuly searchable. Is the index fully searchable? In other words, does it have at least one searchable copy of each bucket? If even one bucket in the index does not have a searchable copy, this field will report the index as non-searchable.
• Searchable Data Copies. The number of complete searchable copies of the index that the cluster has.
• Replicated Data Copies. The number of copies of the index that the cluster has. Each copy must be complete, with no buckets missing.
• Buckets. The number of buckets in the index.
• Cumulative Raw Data Size. The size of the index, excluding hot buckets.

The list of indexes include the internal indexes, _audit and _internal. As you would expect in a cluster, these internal indexes contain the combined data generated by all peers in the cluster. If you need to search for the data generated by a single peer, you can search on the peer's host name.

Search heads tab

For each search head accessing this cluster, the master dashboard lists:

• Search head name. The search head's serverName, as specified in its $SPLUNK_HOME/etc/system/local/server.conf file.
• Site. (For multisite only.) This column displays the site value for each search head.
• Status. Is the search head up?

Note: The list includes the master node as one of the search heads. Although the master has search head capabilities, you should only use those capabilities for debugging purposes. The resources of the master must be dedicated to fulfilling its critical role of coordinating cluster activities. Under no circumstances should the master be employed as a production search head. Also, unlike a dedicated search head, the search head on the master cannot be configured for multi-cluster search; it can only search its own cluster.

To get more information for any search head, click on the arrow to the left of the search head name. These fields appear:

• Location. The search head's server name and port number.
• GUID. The search head's GUID.