Splunk® Enterprise

Managing Indexers and Clusters of Indexers

Download manual as PDF

Download topic as PDF

Implement search affinity in a multisite indexer cluster

One of the key benefits of multisite indexer clustering is that it allows you to configure a cluster so that search heads get search results only from data stored on their local sites. This reduces network traffic while still providing access to the entire set of data, because each site contains a full copy of the data. This benefit is known as search affinity.

For example, say you have two data centers in California, one in San Francisco and the other in Los Angeles. You set up a two-site cluster, with each site corresponding to a data center. Search affinity allows you to reduce long-distance network traffic. Search heads at the San Francisco data center get results only from the peers in San Francisco, while search heads in Los Angeles get results only from their local peers.

How search affinity works

For those sites that you want to support search affinity, you must configure multisite clustering so that the site has a full set of searchable data and a local search head. The search head on any particular site then gets data only from its local site, as long as that site is valid.

When search affinity is functioning, each search head sends its searches to all peers, across all sites, but only the local peers search their data and return results to the search head.

If a local peer holding some of the searchable data goes down and the site temporarily loses its valid state, the search head will, if necessary, get data from peers on remote sites while the local site is undergoing bucket fixing. During this time, the search head will still get as much of the data as possible from the local site.

Once the site regains its valid state, new searches again occur across only the local site.

For more details on how the cluster handles search affinity, see "Multisite indexer cluster architecture" and "Search locally in a multisite cluster".

Implement search affinity

With multisite clusters, search affinity is enabled by default. However, you must perform a few steps to take advantage of it. Specifically, you must ensure that both the searchable data and the search heads are available locally.

To implement search affinity:

1. Configure the site search factor so that you have at least one searchable copy on each site where you require search affinity.

One way to do this is to explicitly specify a search factor for each site that requires search affinity. For example, a four-site cluster with site_search_factor = origin:1, site1:1, site2:1, total:3 ensures that both site1 and site2 have searchable copies of every bucket. The third set of searchable copies will be spread across the two non-explicit sites, with no guarantee that either site will have a full set of searchable copies. Thus, search affinity is enabled for only site1 and site2. Site1 and site2 will each hold primary copies of all buckets.

There are also ways to configure the site search factor to ensure that all sites have searchable copies, even without explicitly specifying some or all of them. For example, a three-site cluster with site_search_factor = origin:1, total:3 guarantees one searchable copy per site, and thus enables search affinity for each site. Each site will have primary copies of all buckets.

For more information on how replication and search factors distribute copies across sites, see "Configure the site replication factor" and "Configure the site search factor".

2. Deploy a search head on each site where you require search affinity.

Disable search affinity

You can disable search affinity for any search head. When search affinity is disabled, the search head does not attempt to obtain search results from a single site only. Rather, it can obtain results from multiple sites. This can be useful, for example, if you have two data centers in close proximity with low latency, and you want to improve overall performance by spreading the processing across indexers on both sites.

What happens when search affinity is disabled

When search affinity is disabled on a search head, search results can come from indexers on any or all sites. If the site search factor stipulates searchable bucket copies on multiple sites, the search head uses undefined criteria to choose which of the searchable copies to search. It is likely to choose some bucket copies from one site and other bucket copies from other sites, so the results will come from multiple sites.

Search heads always select from primary bucket copies. For example, say you have a two site cluster with this search factor:

site_search_factor = origin:2, total:3

The origin site will store two searchable copies and the second site will store one searchable copy of each bucket. So, for some buckets (those originating on site1), site1 will have two searchable copies and for other buckets (those originating on site2), site2 will have two searchable copies. Each site, however, has only a single primary copy.

A search head with search affinity enabled limits its searches to the primary copies on its own site, when possible.

In contrast, a search head with search affinity disabled distributes its search across primary copies on both sites. For a given bucket, you cannot know whether it will select the primary on site1 or the primary on site2. It does tend to use the same primaries from one search to the next.

How to disable search affinity

To disable search affinity for a search head, set the search head's site value to "site0" in server.conf:

[general]
site = site0

[clustering]
multisite = true
...

By setting site=site0, you cause searches to behave like they would on a single-site cluster, with no preference for any particular site.

For more information on configuring multisite search heads, see "Configure the search heads."

PREVIOUS
Multisite indexer cluster deployment overview
  NEXT
Configure multisite indexer clusters with server.conf

This documentation applies to the following versions of Splunk® Enterprise: 6.3.0, 6.3.1, 6.3.2, 6.3.3, 6.3.4, 6.3.5, 6.3.6, 6.3.7, 6.3.8, 6.3.9, 6.3.10, 6.3.11, 6.3.12, 6.3.13, 6.3.14, 6.4.0, 6.4.1, 6.4.2, 6.4.3, 6.4.4, 6.4.5, 6.4.6, 6.4.7, 6.4.8, 6.4.9, 6.4.10, 6.4.11, 6.5.0, 6.5.1, 6.5.1612 (Splunk Cloud only), 6.5.2, 6.5.3, 6.5.4, 6.5.5, 6.5.6, 6.5.7, 6.5.8, 6.5.9, 6.5.10, 6.6.0, 6.6.1, 6.6.2, 6.6.3, 6.6.4, 6.6.5, 6.6.6, 6.6.7, 6.6.8, 6.6.9, 6.6.10, 6.6.11, 6.6.12, 7.0.0, 7.0.1, 7.0.2, 7.0.3, 7.0.4, 7.0.5, 7.0.6, 7.0.7, 7.0.8, 7.0.9, 7.0.10, 7.1.0, 7.1.1, 7.1.2, 7.1.3, 7.1.4, 7.1.5, 7.1.6, 7.1.7, 7.1.8, 7.2.0, 7.2.1, 7.2.2, 7.2.3, 7.2.4, 7.2.5, 7.2.6, 7.3.0


Comments

Esalesapns2 -

Your comment is interesting, as site=site0 usually works when placed in the [general] stanza, as documented. Perhaps there's a bug affecting how the setting is functioning in your particular setup. Could you contact Support so that they can open a case on this issue?

Sgoodman, Splunker
June 3, 2019

My empirical testing determined that you have to set "site = site0" in the "[clustering]" and all "[clustermaster:..]" stanzas to get the search head to search across all sites in a multi-site cluster, not just general. I have three multi-site indexer clusters and had to set it in all of them to remove the affinity.

Esalesapns2
May 21, 2019

Was this documentation topic helpful?

Enter your email address, and someone from the documentation team will respond to you:

Please provide your comments here. Ask a question or make a suggestion.

You must be logged into splunk.com in order to post comments. Log in now.

Please try to keep this discussion focused on the content covered in this documentation topic. If you have a more general question about Splunk functionality or are experiencing a difficulty with Splunk, consider posting a question to Splunkbase Answers.

0 out of 1000 Characters