Configure forwarder monitoring for the DMC

This topic is a step in the procedure for setting up the Distributed Management Console (DMC) in a multi-instance Splunk Enterprise deployment. See "Multi-instance deployment DMC setup steps."

Prerequisites

For several dashboard monitoring panels to work, your forwarders need unique and persistent GUIDs. One way to accomplish this is to clone your forwarder before starting it. A forwarder's GUID is in instance.cfg.

Setup

Follow the setup steps in Splunk Web, at Distributed Management Console > Settings > Forwarder setup.

About time settings

In forwarder setup, you can enable or disable forwarder monitoring and set the data collection interval. Enabling forwarder monitoring runs a scheduled search that populates dmc_forwarder_assets.csv, a lookup file that resides on the DMC node, in $SPLUNK_HOME/etc/apps/splunk_management_console/lookups. The DMC uses this forwarder asset table to know which forwarders to display information about in the forwarder monitoring dashboards.

You can see the scheduled search (but you should not modify it) in Splunk Web in Settings > Searches and reports > DMC Forwarder - Build Asset Table.

On the Distributed Management Console > Settings > Forwarder Monitoring Setup page, you can choose from several values for data collection interval. This interval determines how often that scheduled search runs. The default value is 15 minutes.

When the scheduled search runs to rebuild the forwarder asset table, on whichever schedule you choose, it always looks back 15 minutes. This lookback time is not configurable, and it is different from the data collection interval.

For example, you could set the data collection interval to 24 hours. Then the scheduled search would run once every 24 hours, but it still would check only the 15 minutes before it starts running.

The scheduled search can be expensive, if you have many -- say, hundreds of thousands of -- forwarders. You might find that you want to run the search less often than the default value of every 15 minutes.

Rebuild the forwarder asset table

The data in the forwarder asset table are cumulative. If a forwarder connects to an indexer, its record exists in the table. Then if you later remove the forwarder from your deployment, the forwarder's record is not removed from the asset table. It is instead marked "missing" in the asset table, and it still appears in the DMC forwarder dashboards.

To remove a forwarder entirely from the DMC dashboards, click rebuild forwarder assets in Distributed Management Console > Settings > Forwarder Monitoring Setup. This one time that you run this populating search, you can choose a lookback time. This selection does not change the 15 minute lookback time for the scheduled search or the data collection interval, both discussed above.