General troubleshooting issues

Clock skew between search heads and search peers can affect search behavior

You must keep the clocks on your search heads and search peers in sync, via NTP (network time protocol) or some similar means. The nodes require close clock alignment, so that time comparisons are valid across systems. If the clocks are out-of-sync by more than a few seconds, distributed search cannot work correctly, resulting in search failures or premature expiration of search artifacts.

When you add a search peer to a search head, the search head checks that the clocks are in sync. This check ensures that the system time, independent of the timezone, agrees across the nodes of a distributed search environment. If the nodes are out of sync, the search head rejects the search peer and displays a banner message like this:

The time difference between this system and the intended peer at 
uri=https://servername:8089/ was too big. Please bring the system 
clocks into agreement.

Note: The search head does not run this check if you add the search peer by direct edit of distseach.conf.

Searches can fail if configurations in a knowledge bundle have not yet been replicated to search peers

Configuration changes can take a short time to propagate from search heads to search peers. As a result, during the time between when configuration changes are made on the search head and when they're replicated to the search peers (typically, not more than a few minutes), distributed searches can either fail or provide results based on the previous configuration.

Types of configuration changes that can cause search failures are those that involve new apps or changes to authentication.conf or authorize.conf. Examples include:

• changing the allowed indexes for a role and then running a search as a user within that role
• creating a new app and then running a search from within that app

Any failures will be noted in messages on the search head.

Types of changes that can provide results based on the previous configuration include changing a field extraction or a lookup table file.

To remediate, run the search again.

Network problems can reduce search performance

A 6.0 or later search head by default asks its search peers to generate a remote timeline. This can result in slow searches if the connection between the search head and the search peers is unstable.

The workaround is to add the following setting to limits.conf on the search head :

[search]
remote_timeline_fetchall = false

After making this change, you must restart the search head.