Splunk® Enterprise

Distributed Deployment Manual

Download manual as PDF

Splunk version 4.x reached its End of Life on October 1, 2013. Please see the migration information.
This documentation does not apply to the most recent version of Splunk. Click here for the latest version.
Download topic as PDF

Troubleshoot your deployment

The Deployment Monitor is a great tool for troubleshooting your deployment. Among other features, the monitor's home dashboard includes a set of warnings that provide immediate notice of any abnormalities in your system.

Forwarders behaving badly

The bottom half of the home dashboard provides a number of categories of warnings. Warnings do not necessarily indicate problems in your system, but they do indicate areas that you might need to investigate further.

To get detailed information on any particular warning, click on the green arrow to the right of the warning. For example, you might see a warning that says "5 missing forwarders." To see a list of these missing forwarders, click on the green arrow. You can then drill down within the list for a more detailed look at individual forwarders.

To create an alert that corresponds to a warning, click the Configure Alerting link to its right.

These warnings are just the first step in troubleshooting your deployment. They can indicate serious conditions or, on the other hand, completely benign situations. Depending on your system, there might be excellent reasons for a forwarder to go missing or for other forwarders to be sending less data than expected.

Types of warnings

Warnings indicate possibly unusual behavior in indexers and forwarders. Here are the possible warnings, along with their likely causes and suggested remedies.

Indexer warnings

Warning Likely causes Possible remedies
N idle indexer(s) Possible problem with indexer, network, data sources, or forwarders sending data to the indexer. Drill down into the list of indexers and into the detailed information for each indexer to determine root cause and remedy.
N overloaded indexer(s) Indexer cannot keep up with incoming data flow. This can result in poor search performance and data latency. Add more indexers or filter incoming data.

Forwarder warnings

Warning Likely causes Possible remedies
N missing forwarder(s) Splunk failure, network outage, or issue with underlying system. Trivial causes include laptops or virtualized hosts going on/off line. Drill down into the list of missing forwarders and into the detailed information for each missing forwarder to determine root cause and remedy.
N quiet forwarder(s) Forwarder believes a source has gone silent. Determine whether the problem lies with the source or with the forwarder.
N forwarder(s) sending less than expected Can indicate a problem with the underlying system. Drill down into the list of forwarders and into the detailed information for each forwarder to determine root cause and remedy.
N forwarder(s) sending more than expected Can indicate an attack, data dump due to application crash, or other system problem. Other possibilities include bad Splunk configuration or a new rogue data source configured by a user. Too much data can result in license violations or indexers being unavailable for searches. Drill down into the list of forwarders and into the detailed information for each forwarder to determine root cause and remedy.

Source type warnings

Warning Likely causes Possible remedies
N missing source type(s) A particular source on your forwarders is misconfigured. A source type isn't being correctly applied to a source, perhaps because the source data has changed. Drill down into the list of missing source types and into the detailed information for each missing source type to determine root cause and remedy.
N source type(s) sending less than expected A source type isn't being correctly applied to some of the data from a source. Network issues are preventing data from reaching a Splunk. Drill down into the list of source types and into the detailed information for each source type to determine root cause and remedy.
N source type(s) sending more than expected Data from one source is being incorrectly source-typed. A system is in an error loop and is sending many repeated messages. Drill down into the list of source types and into the detailed information for each source type to determine root cause and remedy.
PREVIOUS
Explore your deployment
  NEXT
Drill for details

This documentation applies to the following versions of Splunk® Enterprise: 4.3, 4.3.1, 4.3.2, 4.3.3, 4.3.4, 4.3.5, 4.3.6, 4.3.7


Was this documentation topic helpful?

Enter your email address, and someone from the documentation team will respond to you:

Please provide your comments here. Ask a question or make a suggestion.

You must be logged into splunk.com in order to post comments. Log in now.

Please try to keep this discussion focused on the content covered in this documentation topic. If you have a more general question about Splunk functionality or are experiencing a difficulty with Splunk, consider posting a question to Splunkbase Answers.

0 out of 1000 Characters