Configure multisite indexer clusters with server.conf
Read this first
Before reading this topic, see:
- "Multisite indexer cluster deployment overview". This topic provides important background information about configuring a multisite cluster.
- "Configure the indexer cluster with server.conf". This topic explains the basics of cluster configuration. It focuses on single-site clusters, but most of the information is relevant to multisite clusters as well.
How multisite configuration differs from single-site configuration
You configure multisite indexer clusters in a similar way to how you configure clusters for a single site, with the exception of a few new attributes:
On all multisite cluster nodes (manager/peers/search heads):
- The
site
attribute specifies the site that a node resides on.
On the manager node and search head:
- The
multisite
attribute indicates that the manager or search head is participating in a multisite cluster.
On the manager node only:
- The
available_sites
attribute names the sites that the manager is managing. - The
site_replication_factor
replaces the standardreplication_factor
used with single-site clusters. For details, see "Configure the site replication factor". - The
site_search_factor
replaces the standardsearch_factor
used with single-site clusters. For details, see "Configure the site search factor".
Important: Although the site_replication_factor
effectively replaces the single-site replication_factor
, and the site_search_factor
replaces single-site search_factor
, those single-site attributes continue to exist in the manager's configuration, with their default values of 3 and 2, respectively. This can cause problems on start-up if any site has fewer peer nodes than either of those values; that is, if any site has only one or two peer nodes. The symptom will be a message that the multisite cluster does not meet its replication or search factor. For example, if one of your sites has only two peers, the default single-site replication factor of 3 will cause the error to occur. To avoid or fix this problem, you must set the single-site replication and search factors to values that are less than or equal to the smallest number of peers on any site. In the case where one site has only two peers, you must therefore explicitly set the replication_factor
attribute to a value of 2. See "Multisite cluster does not meet its replication or search factors."
If you are migrating a cluster from single-site to multisite, you must keep the existing replication_factor
and search_factor
attributes for the existing single-site buckets, while also adding the new multisite site_replication_factor
and site_search_factor
attributes for the new multisite buckets. See "Migrate an indexer cluster from single-site to multisite".
Configure multisite cluster nodes
To configure a multisite cluster, you configure the nodes for each site, editing each node's server.conf
file. For details on the clustering attributes, read the server.conf specification.
Site values
Site values identify the site on which a node resides. You assign a site value to each node in a multisite cluster. To do this, you set the site
attribute in the node's [general]
stanza.
Site values have the syntax:
site<n>
where <n> is an integer in the range of 0 to 63: site1, site2, site3, ....
For example:
site=site1
The special value "site0" can be set only on search heads, or on forwarders that are participating in indexer discovery:
- For a search head, "site0" disables search affinity for the search head. See Disable search affinity.
- For a forwarder participating in indexer discovery, "site0" causes the forwarder to send data to all peer nodes across all sites. See Use indexer discovery in a multisite cluster.
Configure the manager node
You configure the key attributes for the entire cluster on the manager node. Here is an example of a multisite configuration for a manager node:
[general] site = site1 [clustering] mode = manager multisite = true available_sites = site1,site2 site_replication_factor = origin:2,total:3 site_search_factor = origin:1,total:2 pass4SymmKey = whatever cluster_label = cluster1
This example specifies that:
- the instance is located on site1.
- the instance is a cluster manager node.
- the cluster is multisite.
- the cluster consists of two sites: site1 and site2.
- the cluster's replication factor is the default "origin:2,total:3".
- the cluster's search factor is "origin:1,total:2".
- the cluster's security key is "whatever".
- the cluster label is "cluster1."
Note the following:
- You specify the
site
attribute in the[general]
stanza. You specify all other multisite attributes in the[clustering]
stanza. - You can locate the manager node on any site in the cluster, but each cluster has only one manager node.
- You must set
multisite=true
. - You must list all cluster sites in the
available_sites
attribute. - You must set a
site_replication_factor
and asite_search_factor
. For details, see Configure the site replication factor and Configure the site search factor. - The
pass4SymmKey
attribute, which sets the security key, must be the same across all cluster nodes. See Configure the indexer cluster with server.conf for details. - The cluster label is optional. It is useful for identifying the cluster in the monitoring console. See Set cluster labels in Monitoring Splunk Enterprise.
Important: When the manager starts up for the first time, it blocks indexing on the peers until you enable and restart the full replication factor number of peers. For example, given a three-site cluster with "site_replication_factor = origin:2, site1:1, site2:2, site3:3, total:8", the manager blocks indexing until there are at least eight peers in total across all sites, including at least one in site1, two in site2, and three in site3.
Do not restart the manager while it is waiting for the peers to join the cluster. If you do, you will need to restart the peers a second time.
Configure the peer nodes
To configure a peer node in a multisite cluster, you set a site
attribute in the [general]
stanza. All other configuration settings are identical to a peer in a single-site cluster.
Here is an example configuration for a multisite peer node:
[general] site = site1 [replication_port://9887] [clustering] master_uri = https://10.152.31.202:8089 mode = peer pass4SymmKey = whatever
This example specifies that:
- the instance is located in site1. A peer can belong to only a single site.
- the peer will use port 9887 to listen for replicated data streamed from the other peers. You can specify any available, unused port as the replication port. Do not re-use the management or receiving ports.
- the peer's cluster manager node is located at
10.152.31.202:8089
. - the instance is a cluster peer node.
- the security key is "whatever".
Configure the search heads
Multisite search heads can provide search affinity. For information, see "Implement search affinity in a multisite indexer cluster".
To configure a search head in a multisite cluster, you set a site
attribute in the [general]
stanza and a multisite
attribute in the [clustering]
stanza. All other configuration settings are identical to a search head in a single-site cluster.
Here is an example configuration for a multisite search head node:
[general] site = site1 [clustering] multisite = true master_uri = https://10.152.31.202:8089 mode = searchhead pass4SymmKey = whatever
This example specifies that:
- the instance is located in site1. A search head can belong to only a single site per cluster.
- the search head is a member of a multisite cluster.
- the search head's cluster manager is located at
10.152.31.202:8089
. - the instance is a cluster search head.
- the security key is "whatever".
Note: You can also use server.conf
to enable multi-cluster search, in which the search head searches across multiple clusters, multisite or single-site. For searching across multiple multisite clusters, you can configure the search head to be a member of a different site on each cluster. For details, see "Configure multi-cluster search for multisite clusters".
When reconfiguring a search head that is up-and-running, Splunk recommends that you use the CLI command described in "Configure multisite indexer clusters with the CLI", rather than editing server.conf
directly. If you use the CLI, you do not need to restart the search head.
Restart the cluster nodes
After initial configuration
After configuring instances as multisite cluster nodes, you need to restart all of them (manager, peers, and search head) for the changes to take effect. You can do this by invoking the CLI restart
command on each node:
$SPLUNK_HOME/bin/splunk restart
Important: When the manager starts up for the first time, it blocks indexing on the peers until you enable and restart the full replication factor number of peers. For example, given a three-site cluster with "site_replication_factor = origin:2, site1:1, site2:2, site3:3, total:8", the manager blocks indexing until there are at least eight peers in total across all sites, including at least one in site1, two in site2, and three in site3.
Do not restart the manager while it is waiting for the peers to join the cluster. If you do, you will need to restart the peers a second time.
After changing the configuration
On the manager
You must restart the manager after you change any of the following attributes:
multisite
available_sites
site_replication_factor
site_search_factor
After you restart the manager, you must also initiate a rolling-restart of the cluster peers. If you don't, the cluster will be in an indeterminate state. For information on the splunk rolling-restart
command, see "Use rolling restart".
You do not need to restart if you change the site
value on a manager.
On the peers
If you change the site
value on a peer, you must restart it for the change to take effect.
Important: Although you can use the CLI restart
command when you initially enable an instance as a cluster peer node, you should not use it for subsequent restarts. The restart
command is not compatible with index replication once replication has begun. For more information, including a discussion of safe restart methods, see "Restart a single peer".
On the search head
You do not need to restart if you change the site
value on a search head.
Implement search affinity in a multisite indexer cluster | Configure multisite indexer clusters with the CLI |
This documentation applies to the following versions of Splunk® Enterprise: 8.1.0, 8.1.1, 8.1.2, 8.1.3, 8.1.4, 8.1.5, 8.1.6, 8.1.7, 8.1.8, 8.1.9, 8.1.10, 8.1.11, 8.1.12, 8.1.13, 8.1.14
Feedback submitted, thanks!