Distributed Search

General troubleshooting issues

Clock skew between search heads and search peers can affect search behavior

You must keep the clocks on your search heads and search peers in sync, via NTP (network time protocol) or some similar means. If the clocks are out-of-sync by more than a few seconds, you can end up with search failures or premature expiration of search artifacts.

Searches can fail if configurations in a knowledge bundle have not yet been replicated to search peers

Configuration changes can take a short time to propagate from search heads to search peers. As a result, during the time between when configuration changes are made on the search head and when they're replicated to the search peers (typically, not more than a few minutes), distributed searches can either fail or provide results based on the previous configuration.

Types of configuration changes that can cause search failures are those that involve new apps or changes to authentication.conf or authorize.conf. Examples include:

• changing the allowed indexes for a role and then running a search as a user within that role
• creating a new app and then running a search from within that app

Any failures will be noted in messages on the search head.

Types of changes that can provide results based on the previous configuration include changing a field extraction or a lookup table file.

To remediate, run the search again.

Network problems can reduce search performance

A 6.x search head by default asks its search peers to generate a remote timeline. This can result in slow searches if the connection between the search head and the search peers is unstable.

The workaround is to add the following setting to limits.conf on the search head :

[search]
remote_timeline_fetchall = false

After making this change, you must restart the search head.