Distributed Clustered Deployment with SHC - Multisite (M4 / M14)

The following diagram represents a multisite distributed clustered deployment with a search head cluster (SHC) topology:

Architecture overview

The Multi-Site Distributed Clustered Deployment with a Search Head Cluster (SHC) topology provides continuous operation of your Splunk infrastructure for data collection, indexing, and search with one or more stretched search head clusters spanning two or more sites.

It is the most complex validated architecture, designed for deployments that have strict requirements around high availability and disaster recovery.

You use a cluster manager to manage a multisite cluster. If a disaster occurs and a single cluster manager is used, fail it over manually to the disaster recovery (DR) site. Alternatively, you can deploy high availability (HA) cluster managers to automate this process. To learn about HA cluster managers, see High availability deployment: indexer cluster in the Splunk Enterprise Distributed Deployment manual.

Benefits

The benefits of this topology include the following:

• Optimal failover for users if a search node or data center fails. Search artifacts and other runtime knowledge objects are replicated in the SHC. Configure replication carefully to ensure that it happens across sites as the SHC is not site-aware and artifact replication is nondeterministic. To learn about integrating a search head cluster with a multisite indexer cluster, see Integrate a search head cluster with a multisite indexer cluster in the Splunk Enterprise Distributed Deployment manual.

• Possibility to configure search affinity to use the WAN link between sites only if a search cannot be satisfied locally. To learn about search affinity, see Implement search affinity in a multisite indexer cluster in the Splunk Enterprise Managing Indexers and Clusters of Indexers manual.

Limitations

The limitations of this topology include the following:

• Necessity to keep cross-site latency within limits. To learn about the network latency limits, see Network latency limits for clustered deployments in the Splunk Enterprise Capacity Planning manual.
• Necessity to fail the SHC over manually if a minority of cluster members survive. To learn about a failover of the SHC, see Use static captain to recover from loss of majority in the Splunk Enterprise Distributed Search manual.

To ensure the best experience, see Splunk Enterprise service limits in the Splunk Enterprise Capacity Planning manual.

Additional considerations

• Involve Splunk Professional Services to implement this architecture. To learn about Splunk Professional Services, watch this Splunk video, Maximize your data success.
• Customers deploying Splunk Enterprise on cloud service providers like Amazon Web Services (AWS), Google Cloud Platform (GCP), and Microsoft Azure can leverage object store services for SmartStore implementation. See SmartStore system requirements in the Splunk Enterprise Managing Indexers and Clusters of Indexers manual.
• For on-premises customers, Splunk multisite SmartStore is limited to two physical sites, with each site hosted in an on-premises data center. Both sites host active replicated object stores. See Deploy multisite indexer clusters with SmartStore in the Splunk Enterprise Managing Indexers and Clusters of Indexers manual.
• To make sure that users remain on a single search head throughout their session, in front of the SHC members use a third-party network load-balancer that supports sticky sessions. To learn about the network load-balancer, see Use a load balancer with search head clustering in the Splunk Enterprise Distributed Search manual.
• To monitor the health of your Splunk environment, deploy the monitoring console (MC).
• To manage high availability (HA) cluster managers, use an application load balancer. See Use a load balancer to support cluster manager redundancy in the Splunk Enterprise Managing Indexers and Clusters of Indexers manual.
• Activate standby auxiliary services, such as a deployment server (DS), license manager (LM), MC, and search head cluster deployer (SHC-D), if they fail in the primary site.
• Customers considering deployment of Enterprise Security (ES) in an M14 category code should review the guidance for installation of ES in search head cluster environments. Splunk strongly recommends engaging with Splunk Professional Services when deploying ES in a HA/DR environment. See Install Splunk Enterprise Security in a search head cluster environment in the Splunk Enterprise Security Install and Upgrade Splunk Enterprise Security manual.