Distributed Deployment Manual

 


Monitor your deployment

About the Deployment Monitor

NOTE - Splunk version 4.x reached its End of Life on October 1, 2013. Please see the migration information.

This documentation does not apply to the most recent version of Splunk. Click here for the latest version.

About the Deployment Monitor

Splunk deployments can grow to encompass thousands of Splunk instances, including forwarders, indexers, and search heads. Splunk offers a new Deployment Monitor app that helps you to effectively manage medium- to large-scale deployments, keeping track of all your Splunk instances and providing early warning of unexpected or abnormal behavior.

The Deployment Monitor provides chart-rich dashboards and drilldown pages that offer a wealth of information to help you monitor the health of your system. These are some of the things you can monitor:

  • Index throughput over time
  • Number of forwarders connecting to the indexer over time
  • Indexer and forwarder abnormalities
  • Details for individual forwarders and indexers, such as status and forwarding volume over time
  • Source types being indexed by the system
  • License usage

Where to monitor your deployment

You can enable the Deployment Monitor app on any full Splunk instance, but it only makes sense to enable it on the root node of your Splunk deployment. It's only at the root that the Deployment Monitor has visibility into the entire deployment. Consider these typical deployment scenarios:

  • One indexer with many forwarders sending data to it. Enable the Deployment Monitor on the indexer only.
  • One search head connecting to several indexers, each receiving data from many forwarders. Enable the Deployment Monitor on the search head only.
  • Multiple search heads. If you have multiple search heads and you have enabled search head pooling on them, you need to enable the Deployment Monitor on only one search head. (It's best to enable search head pooling before enabling the Deployment Monitor.) If you enable the Deployment Monitor on just a single search head without setting up pooling across all your search heads, you will see no or incomplete data, limited to the indexers communicating with that particular search head.

Once you enable it at the root node, the Deployment Monitor will gather data about your entire deployment. In most cases, no further configuration is needed.

Enable the Deployment Monitor app

The Deployment Monitor app ships with the product in disabled state. To enable it, select the Enable button next to the app on the Splunk Home page:

To disable the Deployment Monitor go to $SPLUNK_HOME/etc/apps/SplunkDeploymentMonitor/local.

Edit the app.conf file:

[install] state = enabled

Change enabled to disabled and restart Splunk.


Splunk home.png


Then, click on the app to view it. This image shows a somewhat compressed view of the top half of the Deployment Monitor's home dashboard (the bottom half of the dashboard contains a set of warnings, useful for quick troubleshooting):


Deployment monitor start page.png


You can also download the Deployment Monitor from Splunk Apps and install it in the usual manner.

Note: The Deployment Monitor app uses several scheduled searches. Some of these searches run on an hourly basis. After you enable the Deployment Monitor, you should wait until a few minutes after the new hour has passed (for example, 14:02, 17:02) before launching it for the first time, to ensure that the scheduled searches have run at least once. This will make a huge difference in its performance.

Populate the Deployment Monitor with historical data

Once you install the Deployment Monitor, it begins populating its summary indexes with current data about your deployment. It continues to add data to its summary indexes going forward. Over time, the additional data will make it possible for you to compare the current state of your system with historical data about it.

If you want to compare your current system state with its previous states right away, without waiting for new data to accumulate, you can tell the monitor to backfill its summary indexes with data for the previous two weeks. This allows you to compare your current state with historical data soon after installing the app.

To populate the Deployment Monitor with data from the previous two weeks:

1. Click the Backfill Data link near the top of the Home dashboard. This takes you to the Manage Data page.

2. Click the Backfill Summary Indexes button. This adds two weeks' worth of data to the Deployment Monitor's summary indexes. A messsage appears, telling you that it will take a while to populate the summary indexes.

3. Once the backfill operation runs its course, you'll be able to view the previous two weeks' activity on the monitor's dashboards.

It only makes sense to perform this procedure once, soon after installing the app.

The second button on the page, Flush and Backfill Summary Indexes deletes all data currently in the Deployment Monitor and replaces it with data from the last two weeks. Use this button only if you run into problems with your Deployment Monitor's data and need to refresh it. It will cause you to lose any deployment data more than two weeks old.

This documentation applies to the following versions of Splunk: 4.2 , 4.2.1 , 4.2.2 , 4.2.3 , 4.2.4 , 4.2.5 , 4.3 , 4.3.1 , 4.3.2 , 4.3.3 , 4.3.4 , 4.3.5 , 4.3.6 , 4.3.7 View the Article History for its revisions.


Comments

Running Splunk 4.3 on RHEL 6, with two indexers and a search head, I do not see "Flush and Backfill Summary Indexes" anywhere. Has it moved?

Gendalia
March 22, 2012

in a distributed environment you also need to manually create all the indexes that this app requires. you should call this out. it would be great if the app could be re-written so it gives the admin the option of specifying an existing index if preferred. adding new indexes means i have to track the space the indexes use and size accordingly.

Tpdcops
October 26, 2011

You must be logged into splunk.com in order to post comments. Log in now.

Was this documentation topic helpful?

If you'd like to hear back from us, please provide your email address:

We'd love to hear what you think about this topic or the documentation as a whole. Feedback you enter here will be delivered to the documentation team.

Feedback submitted, thanks!