Splunk® Enterprise

Distributed Search

Acrobat logo Download manual as PDF

Splunk Enterprise version 6.x is no longer supported as of October 23, 2019. See the Splunk Software Support Policy for details. For information about upgrading to a supported version, see How to upgrade Splunk Enterprise.
This documentation does not apply to the most recent version of Splunk. Click here for the latest version.
Acrobat logo Download topic as PDF

General troubleshooting issues

Clock skew between search heads and search peers can affect search behavior

You must keep the clocks on your search heads and search peers in sync, via NTP (network time protocol) or some similar means. If the clocks are out-of-sync by more than a few seconds, you can end up with search failures or premature expiration of search artifacts.

Searches can fail if configurations in a knowledge bundle have not yet been replicated to search peers

Configuration changes can take a short time to propagate from search heads to search peers. As a result, during the time between when configuration changes are made on the search head and when they're replicated to the search peers (typically, not more than a few minutes), distributed searches can either fail or provide results based on the previous configuration.

Types of configuration changes that can cause search failures are those that involve new apps or changes to authentication.conf or authorize.conf. Examples include:

  • changing the allowed indexes for a role and then running a search as a user within that role
  • creating a new app and then running a search from within that app

Any failures will be noted in messages on the search head.

Types of changes that can provide results based on the previous configuration include changing a field extraction or a lookup table file.

To remediate, run the search again.

Network problems can reduce search performance

A 6.x search head by default asks its search peers to generate a remote timeline. This can result in slow searches if the connection between the search head and the search peers is unstable.

The workaround is to add the following setting to limits.conf on the search head :

remote_timeline_fetchall = false

After making this change, you must restart the search head.

Last modified on 22 July, 2015
Use the DMC to view distributed search status
Handle slow search peers

This documentation applies to the following versions of Splunk® Enterprise: 6.3.0, 6.3.1, 6.3.2, 6.3.3, 6.3.4, 6.3.5, 6.3.6, 6.3.7, 6.3.8, 6.3.9, 6.3.10, 6.3.11, 6.3.12, 6.3.13

Was this documentation topic helpful?

You must be logged into splunk.com in order to post comments. Log in now.

Please try to keep this discussion focused on the content covered in this documentation topic. If you have a more general question about Splunk functionality or are experiencing a difficulty with Splunk, consider posting a question to Splunkbase Answers.

0 out of 1000 Characters