Splunk® Enterprise

Distributed Search

Download manual as PDF

Download topic as PDF

General troubleshooting issues

Clock skew between search heads and search peers can affect search behavior

You must keep the clocks on your search heads and search peers in sync, via NTP (network time protocol) or some similar means. The nodes require close clock alignment, so that time comparisons are valid across systems. If the clocks are out-of-sync by more than a few seconds, distributed search cannot work correctly, resulting in search failures or premature expiration of search artifacts.

When you add a search peer to a search head, the search head checks that the clocks are in sync. This check ensures that the system time, independent of the timezone, agrees across the nodes of a distributed search environment. If the nodes are out of sync, the search head rejects the search peer and displays a banner message like this:

The time difference between this system and the intended peer at 
uri=https://servername:8089/ was too big. Please bring the system 
clocks into agreement.

Note: The search head does not run this check if you add the search peer by direct edit of distseach.conf.

Searches can fail if configurations in a knowledge bundle have not yet been replicated to search peers

Configuration changes can take a short time to propagate from search heads to search peers. As a result, during the time between when configuration changes are made on the search head and when they're replicated to the search peers (typically, not more than a few minutes), distributed searches can either fail or provide results based on the previous configuration.

Types of configuration changes that can cause search failures are those that involve new apps or changes to authentication.conf or authorize.conf. Examples include:

  • changing the allowed indexes for a role and then running a search as a user within that role
  • creating a new app and then running a search from within that app

Any failures will be noted in messages on the search head.

Types of changes that can provide results based on the previous configuration include changing a field extraction or a lookup table file.

To remediate, run the search again.

Network problems can reduce search performance

A 6.0 or later search head by default asks its search peers to generate a remote timeline. This can result in slow searches if the connection between the search head and the search peers is unstable.

The workaround is to add the following setting to limits.conf on the search head :

[search]
remote_timeline_fetchall = false

After making this change, you must restart the search head.

PREVIOUS
Use the DMC to view distributed search status
  NEXT
Handle slow search peers

This documentation applies to the following versions of Splunk® Enterprise: 6.4.0, 6.4.1, 6.4.2, 6.4.3, 6.4.4, 6.4.5, 6.4.6, 6.4.7, 6.4.8, 6.4.9, 6.4.10, 6.4.11, 6.5.0, 6.5.1, 6.5.1612 (Splunk Cloud only), 6.5.2, 6.5.3, 6.5.4, 6.5.5, 6.5.6, 6.5.7, 6.5.8, 6.5.9, 6.5.10, 6.6.0, 6.6.1, 6.6.2, 6.6.3, 6.6.4, 6.6.5, 6.6.6, 6.6.7, 6.6.8, 6.6.9, 6.6.10, 6.6.11, 6.6.12, 7.0.0, 7.0.1, 7.0.2, 7.0.3, 7.0.4, 7.0.5, 7.0.6, 7.0.7, 7.0.8, 7.0.9, 7.0.10, 7.0.11, 7.1.0, 7.1.1, 7.1.2, 7.1.3, 7.1.4, 7.1.5, 7.1.6, 7.1.7, 7.1.8, 7.1.9, 7.2.0, 7.2.1, 7.2.2, 7.2.3, 7.2.4, 7.2.5, 7.2.6, 7.2.7, 7.2.8, 7.2.9, 7.3.0, 7.3.1, 7.3.2, 7.3.3, 8.0.0


Comments

BainM - Thanks for catching that problem with the specified search head version. I've generalized the text so that it covers 6.0 and later versions.

Sgoodman, Splunker
August 8, 2019

In the "Network problems can reduce search performance" header I believe the comment "A 6.x search head by default...." should probably be updated to "A 7.x search head by default", unless this behavior is no longer prevalent in 7.x SH's?

BainM
July 25, 2019

Was this documentation topic helpful?

Enter your email address, and someone from the documentation team will respond to you:

Please provide your comments here. Ask a question or make a suggestion.

You must be logged into splunk.com in order to post comments. Log in now.

Please try to keep this discussion focused on the content covered in this documentation topic. If you have a more general question about Splunk functionality or are experiencing a difficulty with Splunk, consider posting a question to Splunkbase Answers.

0 out of 1000 Characters