Troubleshoot your deployment

The Deployment Monitor is a great tool for troubleshooting your deployment. Among other features, the app's home dashboard includes a set of warnings that provide immediate notice of abnormalities in your system.

Forwarders behaving badly

The bottom half of the home dashboard provides a number of categories of warnings. Warnings do not necessarily indicate problems in your system, but they do indicate areas that you might need to investigate further.

To get detailed information on any particular warning, click on the green arrow to the right of the warning. For example, you might see a warning that says "5 missing forwarders." To see a list of these missing forwarders, click on the green arrow. You can then drill down within the list for a more detailed look at individual forwarders.

To create an alert that corresponds to a warning, click the Configure Alerting link to its right.

These warnings are just the first step in troubleshooting your deployment. They can indicate serious conditions or, on the other hand, completely benign situations. For example, depending on your system, there might be excellent reasons for a forwarder to go missing or for other forwarders to be sending less data than expected.

Types of warnings

Warnings indicate potentially unusual behavior in indexers and forwarders. Here are the possible warnings, along with their likely causes and suggested remedies.

Indexer warnings

Warning	Likely causes	Possible remedies
N Idle indexer(s)	Possible problem with indexer, network, data sources, or forwarders sending data to the indexer.	Drill down into the list of indexers and into the detailed information for each indexer to determine root cause and remedy.
N Overloaded indexer(s)	Indexer cannot keep up with incoming data flow. This can result in poor search performance and data latency.	Add more indexers or filter incoming data.

Note: As with all other warnings and charts in the Deployment Monitor, you can alter the underlying search for the "overloaded indexers" warning to match the needs of your organization. If you want to alter the meaning of this warning, you should edit only the "DM loaded indexers" saved search. If you make this change, you should also edit the "all_indexers.xml" view by adding same arguments to inline searches in the two HiddenSearch modules driving the bottom panel "# indexer(s)".

Forwarder warnings

Warning	Likely causes	Possible remedies
N Missing forwarder(s)	Splunk failure, network outage, or issue with underlying system. Trivial causes include laptops or virtualized hosts going on/off line.	Drill down into the list of missing forwarders and into the detailed information for each missing forwarder to determine root cause and remedy.
N Quiet forwarder(s)	Forwarder believes a source has gone silent.	Determine whether the problem lies with the source or with the forwarder.
N Forwarder(s) sending less than expected	Can indicate a problem with the underlying system.	Drill down into the list of forwarders and into the detailed information for each forwarder to determine root cause and remedy.
N Forwarder(s) sending more than expected	Can indicate an attack, data dump due to application crash, or other system problem. Other possibilities include bad Splunk configuration or a new rogue data source configured by a user. Too much data can result in license violations or indexers being unavailable for searches.	Drill down into the list of forwarders and into the detailed information for each forwarder to determine root cause and remedy.

Source type warnings

Warning	Likely causes	Possible remedies
N Missing source type(s)	A particular source on your forwarders is misconfigured. A source type isn't being correctly applied to a source, perhaps because the source data has changed.	Drill down into the list of missing source types and into the detailed information for each missing source type to determine root cause and remedy.
N Source type(s) sending less than expected	A source type isn't being correctly applied to some of the data from a source. Network issues are preventing data from reaching a Splunk.	Drill down into the list of source types and into the detailed information for each source type to determine root cause and remedy.
N Source type(s) sending more than expected	Data from one source is being incorrectly source-typed. A system is in an error loop and is sending many repeated messages.	Drill down into the list of source types and into the detailed information for each source type to determine root cause and remedy.