Design principles and best practices for deployment tiers

The following tables present design principles and best practices for deployment tiers. Each design principle reinforces one or more pillars of the Splunk Validated Architectures (SVAs): Availability, Performance, Scalability, Security, and Manageability.

Search tier

This table presents design principles and best practices for the search tier:

No.	Design principles and best practices Your requirements determine which principles and practices apply to you.	SVA pillars
1	Keep the search tier close, in network terms, to the indexing tier. Any network delays between the search tier and indexing tier have direct impact on search performance.	Performance
2	Avoid using multiple independent search heads. Independent search heads do not allow sharing of Splunk artifacts created by users. They do not scale well with respect to resource usage across the search tier. If you don't need isolated search head environments, use search head clustering for scaling.	Availability Scalability Security Management
3	Use search head clustering when scaling the search tier. A search head cluster replicates user artifacts across the cluster and allows intelligent scheduling of search workload across all members of the cluster. It also provides a high availability solution.	Availability Scalability
4	Forward all internal logs of search heads to the indexing tier. Store all indexed data in the indexing tier only. It removes the need to provide a high-performing storage in the search head tier and simplifies management. This principle also applies to other Splunk roles.	Performance
5	Consider using LDAP/SAML authentication whenever possible. Centrally managing user identities for authentication purposes simplifies management of your Splunk deployment and increases security.	Performance Manageability
6	Ensure enough cores to cover concurrent search needs. Every search requires a CPU core to execute. If no cores are available to run a search, the search is queued which causes search delays for the user. This principle also applies to the indexing tier.	Security Manageability
7	Use scheduled search time windows when possible, or a smooth scheduled search load. Scheduled searches often run at specific points in time, for example, on the hour, 5, 15, or 30 minutes after the hour, at midnight). Providing a time window that your search can run in helps avoiding search concurrency hotspots.	Availability Performance Scalability
8	Limit the number of distinct search head clusters (SHCs) to avoid overwhelming the indexing tier. A search workload can be managed automatically within a given search head (SH) environment. Independent SHCs might create more concurrent search workload than the indexer tier or search peer can handle. This principle also applies to careful planning of the number of standalone search heads.	Availability Scalability
9	When building SHCs, use an odd number of nodes, for example, 3, 5, or 7. The SHC captain is elected using a majority- based protocol. The odd number of nodes ensures that a SHC can never be split into even numbers of nodes during network failures.	Availability Manageability

Indexing tier

This table presents design principles and best practices for the indexing tier:

No.	Design principles and best practices Your requirements determine which principles and practices apply to you.	SVA pillars
1	Enable parallel pipelines on capable servers to take advantage of available resources. Using parallelization features, you can use available system resources that would otherwise sit idle. Ensure that I/O performance is adequate before you enable ingest parallelization features.	Performance Scalability
2	Consider using NVMe/SSDs for hot and warm volumes and summaries. NVMe and SSDs are available at cost-effective prices. They remove many I/O limitations that often cause unsatisfactory search performance.	Performance Scalability
3	Keep the indexing tier close, in network terms, to the search tier. The lowest possible network latency has a positive effect on the user experience when searching.	Performance
4	Use index replication when high availability (HA) of historical data or reports is needed. Index replication ensures multiple copies of every event in the cluster to protect against a failure of the search peer. By setting a replication factor, adjust the number of copies to match your SLAs.	Availability
5	Ensure data onboarding hygiene by line breaking, timestamp extraction, time zone (TZ), and by defining a source, source type, and host properly and explicitly for each data source. Establish ongoing monitoring by using the monitoring console. Explicit configuration of data sources instead of relying on Splunk autodetection capabilities has proved significantly beneficial for data ingest capacity and indexing latency, especially in high-volume deployments.	Performance Scalability Manageability
6	Consider configuring the setting of batch mode search parallelization on indexers with excess processing power. Configuring search parallelization features can have a significant impact on search performance for certain types of searches. It also ensures using system resources that might otherwise be unused.	Performance Scalability
7	Monitor for balanced data distribution across indexer nodes and search peers. Even distribution of events and data across the search peers is a critical factor influencing search performance and proper enforcement of a data retention policy.	Performance Scalability Manageability
8	Disable the web UI on indexers in distributed clustered deployments. There is no need to access the WebUI directly on indexers.	Performance Security Manageability
9	Consider using Splunk prebuilt technology add-ons (TAs) for well-known data sources rather than building your own configuration to ensure data onboarding hygiene for well-understood data sources. Splunk TAs can provide faster time to value and ensure optimal implementation.	Performance Manageability
10	Monitor critical indexer metrics. Use Splunk monitoring console that provides key performance metrics on how your indexing tier is performing. The metrics apply to CPU, memory usage, and details of internal Splunk components, such as processes, pipelines, queues, and search.	Availability Performance

Management and utility tier

This table presents design principles and best practices for the management and utility tier:

No.	Design principles and best practices Your requirements determine which principles and practices apply to you.	SVA pillars
1	Consider consolidating a license manager (LM), cluster manager (CM), search head cluster deployer (SHC-D), and monitoring console (MC) on a single instance for small environments. These server roles have very little resource demands and are good candidates for colocation. In larger indexer clusters, the CM may require a dedicated server to efficiently manage the cluster.	Manageability
2	Consider a separate instance of a deployment server (DS) for medium to large deployments. If a significant number of forwarders are managed through the DS, the resource needs increase up to the point where you need a dedicated server to maintain the service.	Manageability
3	Consider multiple DSs behind the load balancer (LB) for deployments with a large number of managed systems. Setting up and proper configuring of DSs may require help from Splunk Professional Services.	Availability Scalability
4	Determine whether DS phoneHomeIntervalInSecs can be backed off from the 60-second default. A longer phone home interval has a positive effect on DS scalability.	Scalability
5	Use a dedicated or secured DS to avoid client exploitation through app deployment. Anyone with access to the DS can modify the Splunk configuration managed by that DS, including potentially deployment of malicious applications to forwarder endpoints. Securing this role appropriately is prudent.	Security
6	Use the MC to monitor the health of your deployment and to alert on health issues The monitoring console provides a prebuilt, Splunk set of monitoring solutions and health checks. It also includes extensible platform alerts that can notify you about degrading health of your environment.	Availability Performance Manageability