Configure the distributed management console
What is the distributed management console?
The distributed management console lets you view detailed performance information about your Splunk Enterprise deployment. The topics in this chapter describe the available dashboards and alerts.
The available dashboards provide insight into your deployment's indexing performance, search performance, operating system resource usage, Splunk Enterprise app key value store performance, and license usage.
Find the distributed management console
From anywhere in Splunk Web, click Settings, and then click the Distributed Management Console icon on the left.
The distributed management console (DMC) is visible only to admin users.
You can leave DMC in standalone mode on your Splunk Enterprise instance, which means that you can navigate to the DMC on your individual instance in your deployment and see that particular instance's performance. Or you can go through the configuration steps, still in standalone mode, which lets you access the default platform alerts. Finally, if you go through the configuration steps for distributed mode, you can log into one instance and view performance information for every instance in the deployment.
Which instance should host the console?
After you have configured the DMC in distributed mode, you can navigate to it on only one instance in your deployment and view the console information for your entire deployment.
You have several options for where to host the distributed management console. The instance you choose must be provisioned as a search head. See "Reference hardware" in the Capacity Planning Manual. For security and some performance reasons, only Splunk Enterprise administrators should have access to this instance.
Important: Except for the case of a standalone, non-distributed Splunk Enterprise deployment, the instance hosting the DMC should not be used as a production search head and should not run any searches unrelated to its function as the DMC. This table describes the recommended locations for the DMC, based on deployment type:
|Distributed||Indexer clustering||Search head clustering||DMC options|
|No||N/A||N/A||The standalone instance.|
|Yes||No||No||The license master or a deployment server servicing a small number (<50) of clients. Use of the instance should be limited to DMC and these specific functions. If neither a license master nor a deployment server is available, run the DMC on a dedicated search head not used for other purposes.|
|Yes||Single cluster||Not relevant||The master node. If preferred, you can instead run the DMC on a dedicated search head not used for other purposes.|
|Yes||Multiple clusters||Not relevant||A search head that is configured as a search head node across all the clusters. This search head must be limited only to DMC use.|
|Yes||No||Yes||The search head cluster deployer. If preferred, you can instead run the DMC on a dedicated search head not used for other purposes.|
In a deployment with a single indexer cluster: On the master node
In an indexer cluster, host the DMC on the master node. See "System requirements" in the Managing Indexes and Clusters Manual.
As an alternative, you can host the DMC on a search head node in the cluster. If you do so, however, you cannot use the search head to run any non-DMC searches.
In a deployment with multiple indexer clusters: On a dedicated search head node
If your deployment has multiple indexer clusters, host the DMC on a search head configured as a search head node on each of the clusters. Do not use this search head to run any non-DMC searches.
The main steps to accomplish this are:
1. Configure a single search head as a node on each of the indexer clusters. See Search across multiple indexer clusters" in the Managing Indexes and Clusters Manual. This is your DMC instance.
2. Configure each master node, as well as all search head nodes in the clusters, as search peers of the DMC instance. See Add instances as search peers."
Caution: Do not configure the cluster peer nodes (indexers) as search peers to the DMC node. As nodes in the indexer clusters, they are already known to all search head nodes in their cluster, including the DMC node.
In a non-indexer-cluster environment, option 1: On license master
You can configure the monitoring console on your license master if the following are true:
- Your license master can handle the search workload, that is, meets or exceeds the search head reference hardware requirements. See "Reference hardware" in the Capacity Planning Manual.
- Only Splunk Enterprise admins can access your dedicated license master.
In a non-indexer-cluster environment, option 2: On a new instance
Another option is to provision a new instance, configure it as a search head of search heads and a search head of indexers, and configure the DMC in distributed mode there.
In a search head cluster environment
Use a deployer or dedicated license master for hosting the DMC. The DMC cannot be on a search head cluster member. See "System requirements and other deployment considerations for search head clusters" in the Distributed Search Manual.
The distributed management console is not supported in a search head pooled environment.
The DMC and deployment server
In most cases, you cannot host the distributed DMC on a deployment server. The exception is if the deployment server handles only a small number of deployment clients, no more than 50. The DMC and deployment server functionalities can interfere with each other at larger client counts. See "Deployment server provisioning" in the Updating Splunk Enterprise Instances manual.
Configure your DMC to monitor a deployment
- Have a functional Splunk Enterprise deployment. See "Distributed Splunk Enterprise overview" in the Distributed Deployment Manual. Any instance that you want to monitor must be running Splunk Enterprise 6.1 or higher.
- Check whether your deployment is healthy, that is, that all peers are up.
- Make sure that each instance in the deployment (each search head, license master, and so on) has a unique server.conf
serverNamevalue and inputs.conf
- Forward internal logs (both
$SPLUNK_HOME/var/log/introspection) to indexers from all other instance types. See "Forward search head data" in the Distributed Search Manual. Without this step, many dashboards will lack data. These other instance types include:
- Search heads.
- License masters.
- Cluster masters.
- Deployment servers.
- The user setting up the Distributed Management Console needs the "admin_all_objects" capability.
Add instances as search peers
1. Log into the instance on which you want to configure the distributed management console.
2. In Splunk Web, select Settings > Distributed search > Search peers.
3. Add each search head, deployment server, license master, and standalone indexer as a distributed search peer to the instance hosting the distributed management console. You do not need to add clustered indexers, but you must add clustered search heads.
Set up DMC in distributed mode
1. Log into the instance on which you want to configure the distributed management console. The instance by default is in standalone mode, unconfigured.
2. In Splunk Web, select Distributed management console > Setup.
3. Turn on distributed mode at the top left.
4. Check that:
- The columns labeled instance and machine are populated correctly and populated with values that are unique within a column. Note: If your deployment has nodes running Splunk Enterprise 6.1.x (instead of 6.2.0+), their instance (host) and machine values will not be populated.
- To find the value of machine, typically you can log into the 6.1.x instance and run
hostnameon *nix or Windows. Here machine represents the FQDN of the machine.
- To find the value of instance (host), use btool:
splunk cmd btool inputs list default.
- When you know these values, in the Setup page, click Edit > Edit instance. A popup presents you with two fields to fill in: Instance (host) name and Machine name.
- To find the value of machine, typically you can log into the 6.1.x instance and run
- The server roles are correct, with the primary or major roles. For example, a search head that is also a license master should have both roles marked. If not, click Edit to correct.
- A cluster master is identified if you are using indexer clustering. If not, click Edit to correct.
Caution: Make sure anything marked an indexer is really an indexer.
5. (Optional) Set custom groups. Custom groups are tags that map directly to distributed search groups. You don't need to add groups the first time you go through DMC setup (or ever). You might find groups useful, for example, if you have multisite indexer clustering (each group can consist of the indexers in one location) or an indexer cluster plus standalone peers. Custom groups are allowed to overlap. That is, one indexer can belong to multiple groups. See distributed search groups in the Distributed Search Manual.
6. Click Save.
7. (Optional) Set up platform alerts.
If you add another node to your deployment later, return to Setup and check that the items in step 4 are accurate.
Configure on a single instance
On a single Splunk Enterprise instance operating by itself, you must configure standalone mode before you can use platform alerts.
1. Navigate to the Setup page in DMC.
2. Check that search head, license master, and indexer are listed under Server Roles, and nothing else. If not, click Edit.
3. Click Apply Changes to complete setup.
Use the license usage report view
Return the DMC to default settings
This documentation applies to the following versions of Splunk® Enterprise: 6.2.0, 6.2.1, 6.2.2, 6.2.3, 6.2.4, 6.2.5, 6.2.6, 6.2.7, 6.2.8, 6.2.9, 6.2.10, 6.2.11, 6.2.12, 6.2.13, 6.2.14, 6.2.15