Splunk® Enterprise

Troubleshooting Manual

Download manual as PDF

This documentation does not apply to the most recent version of Splunk. Click here for the latest version.
Download topic as PDF

Too many search jobs

A real-time (all-time) scheduled search will spawn many search jobs, populating the search dispatch directory, when there is no suppression (aka alert throttling). This can negatively effect search performance.

Symptom

Splunk Web displays a yellow banner warning of too many search jobs in the dispatch directory.

Remedies

First, check that for any real-time all-time scheduled searches, you've configured alert throttling. Configure throttling in Settings > Searches and Reports. Read more about throttling in Define per-result alerts in the Alerting Manual.

Already throttled alerts and still getting the warning? A second step you can take is to make alert expiration shorter than the default of 24 hours. If you can, change "alert expiration time" from 24 hours to 1 hour (or less, if you need your alert triggered very frequently). See Manage search jobs in the Search Manual.

The Splunk on Splunk App, version 3.0+, has a helpful view, Dispatch Directory Inspector. The view provides details on search artifacts, including breaking down the disk usage footprint. SoS 3.0.1 and onwards ships with an "as-is" troubleshooting script, dispatch_inspector.py, in etc/apps/sos/bin.

PREVIOUS
I can't find my data!
  NEXT
Dashboard in app is not showing the expected results

This documentation applies to the following versions of Splunk® Enterprise: 6.0, 6.0.1, 6.0.2, 6.0.3, 6.0.4, 6.0.5, 6.0.6, 6.0.7, 6.0.8, 6.0.9, 6.0.10, 6.0.11, 6.0.12, 6.0.13, 6.0.14, 6.0.15, 6.1, 6.1.1, 6.1.2, 6.1.3, 6.1.4, 6.1.5, 6.1.6, 6.1.7, 6.1.8, 6.1.9, 6.1.10, 6.1.11, 6.1.12, 6.1.13, 6.1.14, 6.2.0, 6.2.1, 6.2.2, 6.2.3, 6.2.4, 6.2.5, 6.2.6, 6.2.7, 6.2.8, 6.2.9, 6.2.10, 6.2.11, 6.2.12, 6.2.13, 6.2.14, 6.2.15, 6.3.0, 6.3.1, 6.3.2, 6.3.3, 6.3.4, 6.3.5, 6.3.6, 6.3.7, 6.3.8, 6.3.9, 6.3.10, 6.3.11, 6.3.12, 6.3.13, 6.3.14


Was this documentation topic helpful?

Enter your email address, and someone from the documentation team will respond to you:

Please provide your comments here. Ask a question or make a suggestion.

You must be logged into splunk.com in order to post comments. Log in now.

Please try to keep this discussion focused on the content covered in this documentation topic. If you have a more general question about Splunk functionality or are experiencing a difficulty with Splunk, consider posting a question to Splunkbase Answers.

0 out of 1000 Characters