The reference hardware provides a set of specifications to use when scoping and scaling the Splunk platform. It also sets a baseline for performance when handling search and indexing loads.
Reference machine for single-instance deployments
The following machine requirements represent the basic building block of a Splunk Enterprise deployment.
- Intel x86 64-bit chip architecture
- 12 CPU cores at 2Ghz or greater per core.
- 12GB RAM
- Standard 1Gb Ethernet NIC, optional second NIC for a management network
- Standard 64-bit Linux or Windows distribution
The disk subsystem for a reference machine should be capable of handling a high number of averaged Input/Output Operations Per Second (IOPS).
IOPS are a measurement of how much data throughput a hard drive can produce. Because a hard drive reads and writes at different speeds, there are IOPS numbers for disk reads and writes. The average IOPS is the blend between those two figures.
The more average IOPS a hard drive can produce, the more data it can index and search in a given period of time. While many variable items factor into the amount of IOPS that a hard drive can produce, the following are three most important elements:
- Its rotational speed in revolutions per minute.
- Its average latency, which is the amount of time it takes to spin its platters half a rotation.
- Its average seek time, which is the amount of time it takes to retrieve a requested block of data.
To get the most IOPS out of a hard drive, choose drives that have high rotational speeds and low average latency and seek times. Every drive manufacturer provides this information, and some provide much more.
For information on IOPS and how to calculate them, see the following articles:
- Getting the hang of IOPS (http://www.symantec.com/connect/articles/getting-hang-iops-v13) on Symantec's Connect Community.
- Analyzing I/O performance in Linux (http://www.cmdln.org/2010/04/22/analyzing-io-performance-in-linux) on CMDLN.ORG (A sysadmin blog).
For this application, we use eight 146-gigabyte, 15,000 RPM serial-attached SCSI (SAS) HDs in a Redundant Array of Independent Disks (RAID) 1+0 fault tolerance scheme as the disk subsystem. Each hard drive is capable of about 200 average IOPS. The combined array produces a little over 800 IOPS.
Important: Insufficient disk I/O is the most common limitation found in a Splunk infrastructure. Carefully review the disk subsystem requirements before provisioning your hardware.
Maximum performance capability
The maximum performance capabilities measure indexing and search performance independently, and do not represent the combined load of a typical Splunk use-case. To review performance recommendations for a reference machine with indexing with search load, see Summary of performance recommendations in this manual.
- Up to 20MB per second (1700GB per day) of raw indexing performance if no searching or other index-related activity occurs.
- Up to 50,000 events per second for dense searches.
- Up to 5,000 events per second for sparse searches.
- Up to 2 seconds per index bucket for super-sparse searches.
- From 10 to 50 buckets per second for rare searches with bloom filters.
To find out more about the types of searches and how they affect Splunk Enterprise performance, see How search types affect Splunk Enterprise performance in this manual.
Reference machine for distributed deployments
As the number of active users increases along with the data ingestion rate, the architecture requirements change from a single instance to a distributed Splunk Enterprise environment. The search head and indexer roles have unique hardware recommendations.
Dedicated search head
A search head will utilize CPU resources more consistently than an indexer, but does not require the fast disk throughput or a large pool of local storage for indexing.
- Intel 64-bit chip architecture
- 16 CPU cores at 2Ghz or greater per core.
- 12GB RAM
- 2 x 300GB, 10,000 RPM SAS hard disks, configured in RAID 1
- A 1Gb Ethernet NIC, optional 2nd NIC for a management network
- A 64-bit Linux or Windows distribution
A search request uses 1 CPU core while running. Add additional CPU cores to a search head to accommodate more active users and a higher concurrent search load. You must account for scheduled searches when provisioning a search head. For a review on how searches are prioritized, see the topic Configure the priority of scheduled reports in the Reporting Manual. For information on scaling search performance, see How to maximize search performance in this manual.
Distributing the indexing process allows the Splunk platform to scale data consumption into terabytes a day. A single indexer carries the same disk I/O bandwidth requirements as a group of indexers. Adding additional indexers allows the work of search requests and data indexing to be shared across many instances.
- Intel 64-bit chip architecture.
- 12 CPU cores at 2Ghz or greater per core.
- 12GB RAM.
- Disk subsystem capable of 800 average IOPS. For details, see the topic Disk subsystem.
- A 1Gb Ethernet NIC, with optional second NIC for a management network.
- A 64-bit Linux or Windows distribution.
If the indexer CPU cores exceeds the reference machine specifications, consider implementing the Parallelization settings to improve the indexer performance for specific use cases.
As a guideline for planning your storage infrastructure, indexers do many bulk reads and disk seeks. At higher daily volumes, local disk might not provide cost-effective storage for the time frames where you want a fast search. Optionally, deploy fast attached storage or networked storage, such as storage area networks (SAN) over fiber, that can provide the required IOPS per indexer.
- More disks (specifically, more spindles) are better for indexing performance.
- Total throughput of the entire system is important.
- The ratio of disks to disk controllers in a particular system should be higher, similar to how you provision a database server.
Ratio of indexers to search heads
There is no practical limitation on the number of search heads an indexer can support, or on the number of indexers a search head can search against. The use-case will determine what Splunk instance role (search head or indexer) the infrastructure will need to scale while maintaining performance. For a table with scaling guidelines, see Summary of performance recommendations in this manual.
Premium solutions app requirements
Premium apps can require greater hardware resources than the standard reference machines provide. Before architecting a deployment for a premium app, review the app documentation for scaling and hardware recommendations.
Splunk platform instances are supported in a virtual hosting environment. A VMWare hosted indexer with reserved resources that meet the "reference hardware" specifications can consume data about 10 to 15 percent slower than a indexer hosted on a bare-metal machine. Search performance in a virtual hosting environment is a close match to bare-metal machines.
This describes a best-case scenario that does not account for resource contention with other active virtual machines sharing the same physical server or storage array. It also does not account for certain vendor-specific I/O enhancement techniques, such as Direct I/O or Raw Device Mapping.
For recommendations on running Splunk Enterprise in a VMWare virtual machine, see Deploying Splunk Enterprise Inside Virtual Environments on the main Splunk website.
Splunk Enterprise, self-managed in the cloud
Running Splunk Enterprise in the cloud is an alternative to running it on-premises using bare-metal hardware. Splunk Enterprise delivers similar performance on a cloud-based infrastructure as it does on bare-metal hardware. Depending upon the vendor and technologies used to provision cloud instances, there can be less resources available than the OS reports.
If you run Splunk Enterprise on an Amazon Web Services (AWS) instance:
- AWS measures CPU power on Elastic Compute Cloud (EC2) instances in virtual CPUs (vCPUs), not real CPUs.
- Each vCPU is a hyper thread of an Intel Xeon core on most AWS instance types. See AWS | Amazon EC2 | Instance Types (http://aws.amazon.com/ec2/instance-types/) on the AWS site.
- As a hyper thread of a core, a vCPU acts as a core, but the physical core must schedule its workload among other workloads of other vCPUs that the physical core handles.
For indexing and data storage, note that:
- If you choose to use Elastic Block Storage (EBS), the type of EBS volume you choose determines the amount of performance you get.
- Not all EBS volume types have the necessary IOPS to handle Splunk Enterprise operations.
- The "Provisioned IOPS" and "Magnetic" EBS volume types offer the best opportunity to get the required IOPS necessary for indexing and searching. See EBS - Product Details (http://aws.amazon.com/ebs/details/) on the AWS site.
- Not every EC2 instance type offers the network throughput to the EBS volume that you need. To ensure that bandwidth you must either launch the instance4 as "EBS-optimized" or choose an instance type that provides a minimum of 10Gb of bandwidth. See Amazon EC2 Instance Configuration (http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ebs-ec2-config.html) on the AWS site.
For forwarding, note that the proximity of your cloud infrastructure to your forwarders can have a major impact on performance of the whole environment.
For recommendations on running Splunk Enterprise in AWS, see Deploying Splunk Enterprise On Amazon Webservices on the main Splunk website.
Splunk offers its machine data platform and licensed software as a subscription service called Splunk Cloud. When you subscribe to the service, you purchase a capacity to index, store, and search your machine data. Splunk Cloud abstracts the infrastructure specification from you and delivers high performance on the capacity you have purchased.
Accommodate many simultaneous searches
This documentation applies to the following versions of Splunk® Enterprise: 6.3.0, 6.3.1, 6.3.2, 6.3.3, 6.3.4, 6.3.5, 6.3.6, 6.3.7, 6.3.8, 6.3.9, 6.3.10, 6.3.11, 6.3.12, 6.3.13, 6.3.14, 6.4.0, 6.4.1, 6.4.2, 6.4.3, 6.4.4, 6.4.5, 6.4.6, 6.4.7, 6.4.8, 6.4.9, 6.4.10, 6.4.11