Configure multisite indexer clusters with server.conf

Read this first

Before reading this topic, see:

• "Multisite indexer cluster deployment overview". This topic provides important background information about configuring a multisite cluster.

• "Configure the indexer cluster with server.conf". This topic explains the basics of cluster configuration. It focuses on single-site clusters, but most of the information is relevant to multisite clusters as well.

How multisite configuration differs from single-site configuration

You configure multisite indexer clusters in a similar way to how you configure clusters for a single site, with the exception of a few new attributes:

On all multisite cluster nodes (master/peers/search heads):

• The site attribute specifies the site that a node resides on.

On the master node and search head:

• The multisite attribute indicates that the master or search head is participating in a multisite cluster.

On the master node only:

• The available_sites attribute names the sites that the master is managing.
• The site_replication_factor replaces the standard replication_factor used with single-site clusters. For details, see "Configure the site replication factor".
• The site_search_factor replaces the standard search_factor used with single-site clusters. For details, see "Configure the site search factor".

Important: Although the site_replication_factor effectively replaces the single-site replication_factor, and the site_search_factor replaces single-site search_factor, those single-site attributes continue to exist in the master's configuration, with their default values of 3 and 2, respectively. This can cause problems on start-up if any site has fewer peer nodes than either of those values; that is, if any site has only one or two peer nodes. The symptom will be a message that the multisite cluster does not meet its replication or search factor. For example, if one of your sites has only two peers, the default single-site replication factor of 3 will cause the error to occur. To avoid or fix this problem, you must set the single-site replication and search factors to values that are less than or equal to the smallest number of peers on any site. In the case where one site has only two peers, you must therefore explicitly set the replication_factor attribute to a value of 2. See "Multisite cluster does not meet its replication or search factors."

If you are migrating a cluster from single-site to multisite, you must keep the existing replication_factor and search_factor attributes for the existing single-site buckets, while also adding the new multisite site_replication_factor and site_search_factor attributes for the new multisite buckets. See "Migrate an indexer cluster from single-site to multisite".

Configure multisite cluster nodes

To configure a multisite cluster, you configure the nodes for each site, editing each node's server.conf file. For details on the clustering attributes, read the server.conf specification.

Site values

Site values identify the site on which a node resides. You assign a site value to each node in a multisite cluster. To do this, you set the site attribute in the node's [general] stanza.

Site values have the syntax:

site<n>

where <n> is an integer in the range of 0 to 63: site1, site2, site3, ....

For example:

site=site1

The special value "site0" can be set only on search heads, or on forwarders that are participating in indexer discovery:

• For a search head, "site0" disables search affinity for the search head. See Disable search affinity.
• For a forwarder participating in indexer discovery, "site0" causes the forwarder to send data to all peer nodes across all sites. See Use indexer discovery in a multisite cluster.

Configure the master node

You configure the key attributes for the entire cluster on the master node. Here is an example of a multisite configuration for a master node:

[general]
site = site1

[clustering]
mode = master
multisite = true
available_sites = site1,site2
site_replication_factor = origin:2,total:3
site_search_factor = origin:1,total:2
pass4SymmKey = whatever
cluster_label = cluster1

This example specifies that:

• the instance is located on site1.
• the instance is a cluster master node.
• the cluster is multisite.
• the cluster consists of two sites: site1 and site2.
• the cluster's replication factor is the default "origin:2,total:3".
• the cluster's search factor is "origin:1,total:2".
• the cluster's security key is "whatever".
• the cluster label is "cluster1."

Note the following:

• You specify the site attribute in the [general] stanza. You specify all other multisite attributes in the [clustering] stanza.
• You can locate the master on any site in the cluster, but each cluster has only one master.
• You must set multisite=true.
• You must list all cluster sites in the available_sites attribute.
• You must set a site_replication_factor and a site_search_factor. For details, see Configure the site replication factor and Configure the site search factor.
• The pass4SymmKey attribute, which sets the security key, must be the same across all cluster nodes. See Configure the indexer cluster with server.conf for details.
• The cluster label is optional. It is useful for identifying the cluster in the monitoring console. See Set cluster labels in Monitoring Splunk Enterprise.

Important: When the master starts up for the first time, it blocks indexing on the peers until you enable and restart the full replication factor number of peers. For example, given a three-site cluster with "site_replication_factor = origin:2, site1:1, site2:2, site3:3, total:8", the master blocks indexing until there are at least eight peers in total across all sites, including at least one in site1, two in site2, and three in site3.

Do not restart the master while it is waiting for the peers to join the cluster. If you do, you will need to restart the peers a second time.

Configure the peer nodes

To configure a peer node in a multisite cluster, you set a site attribute in the [general] stanza. All other configuration settings are identical to a peer in a single-site cluster.

Here is an example configuration for a multisite peer node:

[general]
site = site1

[replication_port://9887]

[clustering]
master_uri = https://10.152.31.202:8089
mode = slave
pass4SymmKey = whatever

This example specifies that:

• the instance is located in site1. A peer can belong to only a single site.
• the peer will use port 9887 to listen for replicated data streamed from the other peers. You can specify any available, unused port as the replication port. Do not re-use the management or receiving ports.
• the peer's cluster master is located at 10.152.31.202:8089.
• the instance is a cluster peer ("slave") node.
• the security key is "whatever".

Configure the search heads

Multisite search heads can provide search affinity. For information, see "Implement search affinity in a multisite indexer cluster".

To configure a search head in a multisite cluster, you set a site attribute in the [general] stanza and a multisite attribute in the [clustering] stanza. All other configuration settings are identical to a search head in a single-site cluster. Here is an example configuration for a multisite search head node:

[general]
site = site1

[clustering]
multisite = true
master_uri = https://10.152.31.202:8089
mode = searchhead
pass4SymmKey = whatever

This example specifies that:

• the instance is located in site1. A search head can belong to only a single site per master.
• the search head is a member of a multisite cluster.
• the search head's cluster master is located at 10.152.31.202:8089.
• the instance is a cluster search head.
• the security key is "whatever".

Note: You can also use server.conf to enable multi-cluster search, in which the search head searches across multiple clusters, multisite or single-site. For searching across multiple multisite clusters, you can configure the search head to be a member of a different site on each cluster. For details, see "Configure multi-cluster search for multisite clusters".

When reconfiguring a search head that is up-and-running, Splunk recommends that you use the CLI command described in "Configure multisite indexer clusters with the CLI", rather than editing server.conf directly. If you use the CLI, you do not need to restart the search head.

Restart the cluster nodes

After initial configuration

After configuring instances as multisite cluster nodes, you need to restart all of them (master, peers, and search head) for the changes to take effect. You can do this by invoking the CLI restart command on each node:

$SPLUNK_HOME/bin/splunk restart

Important: When the master starts up for the first time, it blocks indexing on the peers until you enable and restart the full replication factor number of peers. For example, given a three-site cluster with "site_replication_factor = origin:2, site1:1, site2:2, site3:3, total:8", the master blocks indexing until there are at least eight peers in total across all sites, including at least one in site1, two in site2, and three in site3.

Do not restart the master while it is waiting for the peers to join the cluster. If you do, you will need to restart the peers a second time.

After changing the configuration

On the master

You must restart the master after you change any of the following attributes:

• multisite
• available_sites
• site_replication_factor
• site_search_factor

After you restart the master, you must also initiate a rolling-restart of the cluster peers. If you don't, the cluster will be in an indeterminate state. For information on the splunk rolling-restart command, see "Use rolling restart".

You do not need to restart if you change the site value on a master.

On the peers

If you change the site value on a peer, you must restart it for the change to take effect.

Important: Although you can use the CLI restart command when you initially enable an instance as a cluster peer node, you should not use it for subsequent restarts. The restart command is not compatible with index replication once replication has begun. For more information, including a discussion of safe restart methods, see "Restart a single peer".

On the search head

You do not need to restart if you change the site value on a search head.