Splunk® Enterprise

Managing Indexers and Clusters of Indexers

Download manual as PDF

Splunk Enterprise version 5.0 reached its End of Life on December 1, 2017. Please see the migration information.
This documentation does not apply to the most recent version of Splunk. Click here for the latest version.
Download topic as PDF

View the master dashboard

This dashboard provides detailed information on the status of the entire cluster. You can also get information on each of the master's peer nodes from here.

For information on the other clustering dashboards, read:

Access the dashboard

To view the dashboard for the master node:

1. On the master node, click Manager in Splunk Web.

2. In the Distributed Environment group, click Clustering.

You can only view this dashboard on a Splunk instance that has already been enabled as a master.

View the dashboard

The master dashboard contains these sections:

You will also notice a Configure button at the top of the dashboard. You can click the button to reconfigure the settings for the master. However, pay close attention to the warning below.

Warning: Although it is possible to change the settings for the replication factor and search factor, it is inadvisable to increase either of them once your cluster contains significant amounts of data. Doing so will kick off a great deal of bucket activity, which will have an adverse effect on the cluster's performance while bucket copies are being created and/or made searchable.

Master full dashboard.png

Master node details

For the master, the dashboard provides:

  • Master name. The master's serverName, as specified in the peer's $SPLUNK_HOME/etc/system/local/server.conf file.
  • Replication factor. The cluster's replication factor.
  • Search factor. The cluster's search factor.

Cluster overview

The Cluster Overview summarizes the health of your cluster. It tells you:

  • whether the cluster is searchable.
  • how many peers are searchable.
  • how many indexes are searchable.

Depending on the health of your cluster, it might also provide warning messages such as:

  • No peers are configured.
  • A peer is down.
  • The replication factor is not met.
  • The search factor is not met.

For details on the information presented in the Cluster Overview, look at the Peer details and Index details sections of the dashboard.

Peer details

For each peer, the master dashboard lists:

  • Peer Name. The peer's serverName, as specified in the peer's $SPLUNK_HOME/etc/system/local/server.conf file.
  • Searchable. This column indicates whether the peer is currently searchable.
  • Status. The peer's status. For more information about the processes discussed here, read "Take a peer offline". Possible values include:
    • Up
    • Pending. This occurs when a replication fails. It transitions back to Up on the next successful heartbeat from the peer to the master.
    • Detention. A peer goes into detention when it hits resource constraints (for example, it runs out of disk space). While in detention, a peer does not accept inputs and does not participate in searches.
    • Restarting. When you run the offline command, the peer enters this state temporarily after it leaves the ReassigningPrimaries state. It remains in this state for the restart_timeout period (ten minutes by default). If you do not restart the peer within this time, it then moves to the Down state. The peer also enters this state during rolling restarts or if restarted via Splunk Web.
    • ShuttingDown. The master detects that the peer is shutting down.
    • ReassigningPrimaries. A peer enters this state temporarily when you run the offline command.
    • Down. The peer enters this state when it goes offline: either you ran the version of the offline command and the peer shut down for longer than the restart_timeout period (ten minutes by default), or the peer went offline for some other reason (for instance, it crashed).
  • Buckets. The number of buckets for which the peer has copies.

To get detailed information about any peer, click on the peer name. This takes you to the master version of the peer dashboard, described below.

Index details

This section of the dashboard lists the cluster's indexes. For each index, it provides these fields:

  • Index. The name of the index. Internal indexes are preceded by an underscore (_).
  • Searchable. Is the index searchable? In other words, does it have at least one searchable copy of each bucket? If even one bucket in the index does not have a searchable copy, this field will report the index as non-searchable.
  • Replicated Copies. The number of copies of the index that the cluster has. Each copy must be complete, with no buckets missing. A warning icon appears if the number of copies is less than the cluster's replication factor.
  • Searchable Copies. The number of complete searchable copies of the index that the cluster has. A warning icon appears if the number of copies is less than the cluster's searchable factor. For example, if the search factor is 2 and a single searchable bucket is missing from two otherwise searchable copies, the warning icon will appear.
  • Buckets. The number of buckets in the index.
  • Size. The size of the index, excluding hot buckets.

The list of indexes include the internal indexes, _audit and _internal. As you would expect in a cluster, these internal indexes contain the combined data generated by all peers in the cluster. If you need to search for the data generated by a single peer, you can search on the peer's host name.

Access the master view of the peer node dashboard

This dashboard provides detailed information on the status of a peer node. You can access versions of this dashboard from two locations:

  • On the master node. Click on a peer name in the master dashboard, as described in the previous section.
  • On the peer node itself. See "View the peer dashboard".

Here's how the master's version of the peer dashboard looks:

Master view of peer dashboard.png

The dashboard provides information on the peer's status:

  • Location. The peer's IP address and port number.
  • Last heartbeat. The time of the last heartbeat the master received from the peer.
  • Replication port. The port on which the peer receives replicated data from other peers.
  • Status. The peer's status. For more information about the processes discussed here, read "Take a peer offline". Possible values include:
    • Up
    • Pending. This occurs when the master does not receive three consecutive heartbeats from the peer. This is a temporary state, and a peer rarely stays in it for long.
    • Detention. A peer goes into detention when it hits resource constraints (for example, it runs out of disk space). While in detention, a peer does not accept inputs but it can still participate in searches.
    • Restarting. When you run the offline command, the peer enters this state temporarily after it leaves the ReassigningPrimaries state. It remains in this state for the restart_timeout period (ten minutes by default). If you do not restart the peer within this time, it then moves to the Down state. The peer also enters this state during rolling restarts or if restarted via Splunk Web.
    • ShuttingDown. The master detects that the peer is shutting down.
    • ReassigningPrimaries. A peer enters this state temporarily when you run the offline command.
    • Down. The peer enters this state when it goes offline: either you ran the version of the offline command and the peer shut down for longer than the restart_timeout period (ten minutes by default), or the peer went offline for some other reason (for instance, it crashed).
PREVIOUS
Configure the cluster with the CLI
  NEXT
View the peer dashboard

This documentation applies to the following versions of Splunk® Enterprise: 5.0, 5.0.1, 5.0.2, 5.0.3, 5.0.4, 5.0.5, 5.0.6, 5.0.7, 5.0.8, 5.0.9, 5.0.10, 5.0.11, 5.0.12, 5.0.13, 5.0.14, 5.0.15, 5.0.16, 5.0.17, 5.0.18


Was this documentation topic helpful?

Enter your email address, and someone from the documentation team will respond to you:

Please provide your comments here. Ask a question or make a suggestion.

You must be logged into splunk.com in order to post comments. Log in now.

Please try to keep this discussion focused on the content covered in this documentation topic. If you have a more general question about Splunk functionality or are experiencing a difficulty with Splunk, consider posting a question to Splunkbase Answers.

0 out of 1000 Characters