View the master dashboard
This dashboard provides detailed information on the status of the entire indexer cluster. You can also get information on each of the master's peer nodes from here.
For information on the other clustering dashboards, read:
Access the master dashboard
To view the dashboard for the master node:
1. Click Settings on the upper right side of Splunk Web.
2. In the Distributed Environment group, click Indexer clustering.
You can only view this dashboard on an instance that has been enabled as a master.
View the master dashboard
The master dashboard contains these sections:
Here is how the dashboard initially appears, with the Peers tab open:
The cluster overview summarizes the health of your cluster. It tells you:
- whether the cluster's data is fully searchable; that is, whether all buckets in the cluster have a primary copy.
- whether the search and replication factors have been met.
- how many peers are searchable.
- how many indexes are searchable.
Depending on the health of your cluster, it might also provide warning messages such as:
- Some data is not searchable.
- Replication factor not met.
- Search factor not met.
For details on the information presented in the cluster overview, browse the tabs underneath.
On the upper right side of the dashboard, there are three buttons:
- Edit. For information on this button, see "Configure the master with the dashboard.". This button is disabled for multisite clusters.
- More Info. This button provides details on the master node configuration:
For each peer, the master dashboard lists:
- Peer Name. The peer's
serverName, as specified in the peer's
- Fully searchable. This column indicates whether the peer currently has a complete set of primaries and is fully searchable.
- Site. (For multisite only.) This column displays the site value for each peer.
- Status. The peer's status. For more information about the processes discussed here, read "Take a peer offline". Possible values include:
Pending. This occurs when a replication fails. It transitions back to Up on the next successful heartbeat from the peer to the master.
Detention. A peer goes into detention when it hits resource constraints (for example, it runs out of disk space). While in detention, a peer does not accept inputs and does not participate in searches.
Restarting. When you run the
offlinecommand, the peer enters this state temporarily after it leaves the
ReassigningPrimariesstate. It remains in this state for the
restart_timeoutperiod (60 seconds by default). If you do not restart the peer within this time, it then moves to the
Downstate. The peer also enters this state during rolling restarts or if restarted via Splunk Web.
ShuttingDown. The master detects that the peer is shutting down.
ReassigningPrimaries. A peer enters this state temporarily when you run the
Down. The peer enters this state when it goes offline: either you ran the version of the
offlinecommand and the peer shut down for longer than the
restart_timeoutperiod (60 seconds by default), or the peer went offline for some other reason (for instance, it crashed).
- Buckets. The number of buckets for which the peer has copies.
To get more information for any peer, click on the arrow to the left of the peer name. These fields appear:
- Location. The peer's IP address and port number.
- Last Heartbeat. The time of the last heartbeat the master received from the peer.
- Replication port. The port on which the peer receives replicated data from other peers.
- Base generation ID. The peer's base generation ID, which is equivalent to the cluster's generation ID at the moment that the peer last joined the cluster. This ID will be less or equal to the cluster's current generation ID. So, if a peer joined the cluster at generation 1 and has stayed in the cluster ever since, its base generation ID remains 1, even though the cluster might have incremented its current generation ID to, say, 5.
- GUID. The peer's GUID.
Note: After a peer goes down, it continues to appear on the list of peers, although its status changes to "Down" or "GracefulShutdown." To remove the peer from the master's list, see "Remove a peer from the master's list."
For each index, the master dashboard lists:
- Index Name. The name of the index. Internal indexes are preceded by an underscore (_).
- Fully searchable. Is the index fully searchable? In other words, does it have at least one searchable copy of each bucket? If even one bucket in the index does not have a searchable copy, this field will report the index as non-searchable.
- Searchable Data Copies. The number of complete searchable copies of the index that the cluster has.
- Replicated Data Copies. The number of copies of the index that the cluster has. Each copy must be complete, with no buckets missing.
- Buckets. The number of buckets in the index.
- Cumulative Raw Data Size. The size of the index, excluding hot buckets.
The list of indexes include the internal indexes, _audit and _internal. As you would expect in a cluster, these internal indexes contain the combined data generated by all peers in the cluster. If you need to search for the data generated by a single peer, you can search on the peer's host name.
This tab also reveals a button with the label, Bucket Status. If you click on it, you go to the Bucket Status dashboard. See "Bucket status dashboard.".
Note: A new index appears here only after it contains some data. In other words, if you configure a new index on the peer nodes, a row for that index appears only after you send data to that index.
Search heads tab
For each search head accessing this cluster, the master dashboard lists:
- Search head name. The search head's
serverName, as specified in its
- Site. (For multisite only.) This column displays the site value for each search head.
- Status. Is the search head up or down? The master decides that a search head is down if the search head does not poll the master for generation information within a period twice the length of the
generation_poll_interval. That attribute is configurable in
Note: The list includes the master node as one of the search heads. Although the master has search head capabilities, you should only use those capabilities for debugging purposes. The resources of the master must be dedicated to fulfilling its critical role of coordinating cluster activities. Under no circumstances should the master be employed as a production search head. Also, unlike a dedicated search head, the search head on the master cannot be configured for multi-cluster search; it can only search its own cluster.
To get more information for any search head, click on the arrow to the left of the search head name. These fields appear:
- Location. The search head's server name and port number.
- GUID. The search head's GUID.
View the bucket status dashboard
The Bucket Status dashboard provides status for the buckets in the cluster. It contains three tabs:
- Fixup Tasks - In Progress
- Fixup Tasks - Pending
- Indexes with Excess Buckets
Fixup Tasks - In Progress
This tab provides a list of buckets that are currently being fixed. For example, if a bucket has too few copies, fixup activities must occur to return the cluster to a valid and complete state. While those activities are occurring, the bucket appears on this list.
Fixup Tasks - Pending
This tab provides a list of buckets that are waiting to be fixed. You can filter the fixup tasks by search factor, replication factor, and generation.
For more information on bucket fixup activities, see "What happens when a peer goes down."
Indexes with Excess Buckets
This tab provides a list of indexes with excess bucket copies. It enumerates both buckets with excess copies and buckets with excess searchable copies. It also enumerates the total excess copies in each category. For example, if your index "new" has one bucket with three excess copies, one of which is searchable, and a second bucket with one excess copy, which is non-searchable, the row for "new" will report:
- 2 buckets with excess copies
- 1 bucket with excess searchable copies
- 4 total excess copies
- 1 total excess searchable copies
If you want to remove the excess copies for a single index, click the Remove button on the right side of the row for that index.
If you want to remove the excess copies for all indexes, click the Remove All Excess Buckets button.
For more information on excess bucket copies, see "Remove excess bucket copies from the indexer cluster."
Use the distributed management console to view status
You can use the distributed management console (DMC) to monitor most aspects of your deployment, including the status of your indexer cluster. The information available through the console duplicates much of the information available on the master dashboard.
For more information, see "Use the DMC to view indexer cluster status."
Migrate an indexer cluster from single-site to multisite
View the peer dashboard
This documentation applies to the following versions of Splunk® Enterprise: 6.3.0, 6.3.1, 6.3.2, 6.3.3, 6.3.4, 6.3.5, 6.3.6, 6.3.7, 6.3.8, 6.3.9, 6.3.10, 6.3.11, 6.3.12, 6.3.13, 6.3.14