About the Deployment Monitor
Splunk deployments can grow to encompass thousands of Splunk instances, including forwarders, indexers, and search heads. Splunk offers a new Deployment Monitor app that helps you to effectively manage medium- to large-scale deployments, keeping track of all your Splunk instances and providing early warning of unexpected or abnormal behavior.
The Deployment Monitor provides chart-rich dashboards and drilldown pages that offer a wealth of information to help you monitor the health of your system. These are some of the things you can monitor:
- Index throughput over time
- Number of forwarders connecting to the indexer over time
- Indexer and forwarder abnormalities
- Details for individual forwarders and indexers, such as status and forwarding volume over time
- Source types being indexed by the system
- License usage
Where to monitor your deployment
You can enable the Deployment Monitor app on any full Splunk instance, but it only makes sense to enable it on the root node of your Splunk deployment. It's only at the root that the Deployment Monitor has visibility into the entire deployment. Consider these typical deployment scenarios:
- One indexer with many forwarders sending data to it. Enable the Deployment Monitor on the indexer only.
- One search head connecting to several indexers, each receiving data from many forwarders. Enable the Deployment Monitor on the search head only.
- Multiple search heads. If you have multiple search heads and you have enabled search head pooling on them, you need to enable the Deployment Monitor on only one search head. (It's best to enable search head pooling before enabling the Deployment Monitor.) If you enable the Deployment Monitor on just a single search head without setting up pooling across all your search heads, you will see no or incomplete data, limited to the indexers communicating with that particular search head.
Once you enable it at the root node, the Deployment Monitor will gather data about your entire deployment. In most cases, no further configuration is needed.
Enable the Deployment Monitor app
The Deployment Monitor app ships with the product in disabled state. To enable it, select the Enable button next to the app on the Splunk Home page:
To disable the Deployment Monitor go to $SPLUNK_HOME/etc/apps/SplunkDeploymentMonitor/local.
Edit the app.conf file:
[install] state = enabled
Change enabled to disabled and restart Splunk.
Then, click on the app to view it. This image shows a somewhat compressed view of the top half of the Deployment Monitor's home dashboard (the bottom half of the dashboard contains a set of warnings, useful for quick troubleshooting):
You can also download the Deployment Monitor from Splunk Apps and install it in the usual manner.
Note: The Deployment Monitor app uses several scheduled searches. Some of these searches run on an hourly basis. After you enable the Deployment Monitor, you should wait until a few minutes after the new hour has passed (for example, 14:02, 17:02) before launching it for the first time, to ensure that the scheduled searches have run at least once. This will make a huge difference in its performance.
Populate the Deployment Monitor with historical data
Once you install the Deployment Monitor, it begins populating its summary indexes with current data about your deployment. It continues to add data to its summary indexes going forward. Over time, the additional data will make it possible for you to compare the current state of your system with historical data about it.
If you want to compare your current system state with its previous states right away, without waiting for new data to accumulate, you can tell the monitor to backfill its summary indexes with data for the previous two weeks. This allows you to compare your current state with historical data soon after installing the app.
To populate the Deployment Monitor with data from the previous two weeks:
1. Click the Backfill Data link near the top of the Home dashboard. This takes you to the Manage Data page.
2. Click the Backfill Summary Indexes button. This adds two weeks' worth of data to the Deployment Monitor's summary indexes. A messsage appears, telling you that it will take a while to populate the summary indexes.
3. Once the backfill operation runs its course, you'll be able to view the previous two weeks' activity on the monitor's dashboards.
It only makes sense to perform this procedure once, soon after installing the app.
The second button on the page, Flush and Backfill Summary Indexes deletes all data currently in the Deployment Monitor and replaces it with data from the last two weeks. Use this button only if you run into problems with your Deployment Monitor's data and need to refresh it. It will cause you to lose any deployment data more than two weeks old.
Troubleshoot distributed search
Explore your deployment