View the master dashboard
This dashboard provides detailed information on the status of the entire cluster. You can also get information on each of the master's peer nodes from here.
For information on the other clustering dashboards, read:
Access the dashboard
To view the dashboard for the master node:
1. Click Settings in the upper right corner of Splunk Web on the master.
2. In the Distributed environment group, click Clustering.
You can only view this dashboard on an instance that has already been enabled as a master.
View the dashboard
The master dashboard contains these sections:
Here's how the dashboard initially appears, with the Peers tab open:
The Cluster Overview summarizes the health of your cluster. It tells you:
- whether the cluster's data is fully searchable; that is, whether all buckets in the cluster have a primary copy.
- whether the search and replication factors have been met.
- how many peers are searchable.
- how many indexes are searchable.
Depending on the health of your cluster, it might also provide warning messages such as:
- Some data is not searchable.
- Replication factor not met.
- Search factor not met.
For details on the information presented in the Cluster Overview, look through the tabs below the overview.
On the upper right side of the dashboard, there are three buttons:
- Edit. For information on this button, see "Perform edits on the master".
- More Info. This button provides details on the master node configuration:
For each peer, the master dashboard lists:
- Peer Name. The peer's
serverName, as specified in the peer's
- Searchable. This column indicates whether the peer is currently searchable.
- Status. The peer's status. For more information about the processes discussed here, read "Take a peer offline". Possible values include:
Pending. This occurs when a replication fails. It transitions back to Up on the next successful heartbeat from the peer to the master.
Detention. A peer goes into detention when it hits resource constraints (for example, it runs out of disk space). While in detention, a peer does not accept inputs and does not participate in searches.
Restarting. When you run the
offlinecommand, the peer enters this state temporarily after it leaves the
ReassigningPrimariesstate. It remains in this state for the
restart_timeoutperiod (60 seconds by default). If you do not restart the peer within this time, it then moves to the
Downstate. The peer also enters this state during rolling restarts or if restarted via Splunk Web.
ShuttingDown. The master detects that the peer is shutting down.
ReassigningPrimaries. A peer enters this state temporarily when you run the
Down. The peer enters this state when it goes offline: either you ran the version of the
offlinecommand and the peer shut down for longer than the
restart_timeoutperiod (60 seconds by default), or the peer went offline for some other reason (for instance, it crashed).
- Buckets. The number of buckets for which the peer has copies.
To get more information for any peer, click on the arrow to the left of the peer name. These fields appear:
- Location. The peer's IP address and port number.
- Last Heartbeat. The time of the last heartbeat the master received from the peer.
- Replication port. The port on which the peer receives replicated data from other peers.
- Generation ID. The peer's base generation ID, which is equivalent to the cluster's generation ID at the moment that the peer last joined the cluster. This ID will be less or equal to the cluster's current generation ID. So, if a peer joined the cluster at generation 1 and has stayed in the cluster ever since, its base generation ID remains 1, even though the cluster might have incremented its current generation ID to, say, 5.
- GUID. The peer's GUID.
For each index, the master dashboard lists:
- Index Name. The name of the index. Internal indexes are preceded by an underscore (_).
- Searchable. Is the index searchable? In other words, does it have at least one searchable copy of each bucket? If even one bucket in the index does not have a searchable copy, this field will report the index as non-searchable.
- Searchable Data Copies. The number of complete searchable copies of the index that the cluster has.
- Replicated Data Copies. The number of copies of the index that the cluster has. Each copy must be complete, with no buckets missing.
- Buckets. The number of buckets in the index.
- Cumulative Raw Data Size. The size of the index, excluding hot buckets.
The list of indexes include the internal indexes, _audit and _internal. As you would expect in a cluster, these internal indexes contain the combined data generated by all peers in the cluster. If you need to search for the data generated by a single peer, you can search on the peer's host name.
Search heads tab
For each search head accessing this cluster, the master dashboard lists:
- Search Head Name. The search head's
serverName, as specified in the peer's
- Status. Is the search head up?
Note: The list includes the master node as one of the search heads. While the master has search head capabilities, you should only use those capabilities for debugging purposes. The resources of the master must be dedicated to fulfilling its critical role of coordinating cluster activities. Under no circumstances should the master be employed as a production search head. Also, unlike a dedicated search head, the search head on the master cannot be configured for multi-cluster search; it can only search its own cluster.
To get more information for any search head, click on the arrow to the left of the search head name. These fields appear:
- Location. The search head's server name and port number.
- GUID. The search head's GUID.
Perform edits on the master
The Edit button at the top right of the master dashboard offers several options:
- Node Type. Change the instance's node type. Warning: It is extremely unlikely that you will want to change the node type for nodes in an active cluster. Consider the consequences carefully before doing so.
- Master Node Configuration. Change these master node settings:
- Replication Factor. Change the cluster's replication factor. Warning: It is inadvisable to increase the replication factor once your cluster contains significant amounts of data. Doing so will kick off a great deal of bucket activity, which will have an adverse effect on the cluster's performance while bucket copies are being created.
- Search Factor. Change the cluster's search factor. Warning: It is inadvisable to increase the replication factor once your cluster contains significant amounts of data. Doing so will kick off a great deal of bucket activity, which will have an adverse effect on the cluster's performance while bucket copies are being made searchable.
- Security Key. Change the security key. Only change the secret key if you are also changing it for all other nodes in the cluster. The key must be the same across all instances in a cluster.
- Distribute Configuration Bundle. Distribute updated configurations and apps to the set of peer nodes. For details of this process, see the topic "Update common peer configurations and apps".
- Disable Clustering. Remove this node from the cluster. Warning: If you remove the master node from the cluster, the entire cluster will eventually fail.
Configure the cluster with the CLI
View the peer dashboard
This documentation applies to the following versions of Splunk® Enterprise: 6.0, 6.0.1, 6.0.2, 6.0.3, 6.0.4, 6.0.5, 6.0.6, 6.0.7, 6.0.8, 6.0.9, 6.0.10, 6.0.11, 6.0.12, 6.0.13, 6.0.14, 6.0.15