A reference machine helps you understand when to scale and distribute a Splunk Enterprise deployment. Refer to this configuration as the standard for the remainder of this chapter and whenever you need to size the hardware needs of your environment.
Reference machine for single-instance deployments
The following machine requirements represent the basic building block of a Splunk Enterprise deployment.
- Intel x86 64-bit chip architecture
- 2 CPUs, 6 cores per CPU (12 CPU cores total), at least 2Ghz per core
- 12GB RAM
- Standard 1Gb Ethernet NIC, optional second NIC for a management network
- Standard 64-bit Linux or Windows distribution
The disk subsystem for a reference machine should be capable of handling a high number of averaged Input/Output Operations Per Second (IOPS).
IOPS are a measurement of how much data throughput a hard drive can produce. Because a hard drive reads and writes at different speeds, there are IOPS numbers for disk reads and writes. The average IOPS is the blend between those two figures.
The more average IOPS a hard drive can produce, the more data it can index and search in a given period of time. While many variable items factor into the amount of IOPS that a hard drive can produce, the following are three most important elements:
- Its rotational speed in revolutions per minute.
- Its average latency, which is the amount of time it takes to spin its platters half a rotation.
- Its average seek time, which is the amount of time it takes to retrieve a requested block of data.
To get the most IOPS out of a hard drive, choose drives that have high rotational speeds and low average latency and seek times. Every drive manufacturer provides this information, and some provide much more.
For information on IOPS and how to calculate them, see the following articles:
- "Getting the hang of IOPS (http://www.symantec.com/connect/articles/getting-hang-iops-v13) on Symantec's Connect Community.
- "Analyzing I/O performance in Linux (http://www.cmdln.org/2010/04/22/analyzing-io-performance-in-linux) on CMDLN.ORG (A sysadmin blog).
For this application, we use eight 146-gigabyte, 15,000 RPM serial-attached SCSI (SAS) HDs in a Redundant Array of Independent Disks (RAID) 1+0 fault tolerance scheme as the disk subsystem. Each hard drive is capable of about 200 average IOPS. The combined array produces a little over 800 IOPS.
Important: Disk I/O often constrains Splunk Enterprise first, so always consider disk infrastructure first when specifying your hardware.
The reference machine produces the following index and search performance metrics for a given sample of data.
- Up to 20MB per second (1700GB per day) of raw indexing performance, as long as no other Splunk Enterprise related activity occurs.
- Up to 50,000 events per second for dense searches.
- Up to 5,000 events per second for sparse searches.
- Up to 2 seconds per index bucket for super-sparse searches.
- From 10 to 50 buckets per second for rare searches with bloom filters.
To find out more about the types of searches and how they affect Splunk Enterprise performance, see "How search types affect Splunk Enterprise performance" in this manual.
Reference machine for distributed deployments
At higher data indexing rates and user counts, consider the different needs of indexers and search heads. Dedicated search heads do not need extremely fast disk throughput, nor do they need much local storage. They require far more CPU resources than do indexers.
The following recommendations are for both search heads and indexers in a distributed deployment.
Dedicated search head
- Intel 64-bit chip architecture
- 4 CPUs, 4 cores per CPU (16 CPU cores total), at least 2Ghz per core
- 12GB RAM
- 2 x 300GB, 10,000 RPM SAS hard disks, configured in RAID 1
- Standard 1Gb Ethernet NIC, optional 2nd NIC for a management network
- Standard 64-bit Linux or Windows distribution
Because search heads require raw computing power and are likely to become CPU-bound, add additional CPU cores to a search head for faster performance per server. If search workload dictates, a search head might require 4 CPUs with 6 cores per CPU. Regardless, the guideline of 1 core per active user applies. Additionally:
- Account for scheduled searches in your CPU allowance. A typical search request requires 1 CPU core.
- Depending on the type of search you use, you might need to add CPU cores to account for the increased load those search types cause.
Indexers in a distributed deployment configuration have equivalent disk I/O bandwidth requirements to indexers in a single-instance deployment. To compensate for the increase in search requests and higher data indexing volume in a distributed environment, distribute the work across many indexers.
- Intel 64-bit chip architecture.
- 2 CPUs, 6 cores per CPU (12 CPU cores total), at least 2Ghz per core.
- 12GB RAM.
- Disk subsystem capable of 800 average IOPS. See "Disk subsystem."
- Standard 1Gb Ethernet NIC, with optional second NIC for a management network.
- Standard 64-bit Linux or Windows distribution.
At higher daily volumes, local disk does not provide cost-effective storage for the time frames where you want a fast search. In these cases, deploy fast attached storage or networked storage, such as storage area networks (SAN) over fiber, that can provide the required IOPS per indexer. While there are too many types of storage to recommend, consider these guidelines when you plan your storage infrastructure:
- Indexers do many bulk reads.
- Indexers do many disk seeks.
- More disks (specifically, more spindles) are better for indexing performance.
- Total throughput of the entire system is important.
- The ratio of disks to disk controllers in a particular system should be higher, similar to how you configure a database server.
Ratio of indexers to search heads
Technically, there is no practical limitation on the number of search heads an indexer can support or on the number of indexers a search head can search against. However, systems limitations suggest a ratio of approximately 8 indexers to 1 search head as a rough guideline. If you have many searchers compared to your total data volume, then more search heads will increase search efficiency. In some cases, the best use of a separate search head is to populate summary indexes. This search head acts like an indexer to the primary search head that users log into.
Splunk Enterprise performs fastest when deployed directly on to bare-metal hardware. However, Splunk Enterprise can deliver on virtual equipment, and Splunk supports deploying it that way.
Using the bare-metal hardware as a baseline and VMWare as an example, Splunk Enterprise generally indexes data about 10 to 15 per cent slower on a virtual machine than it does on a standard bare-metal machine. Search performance is on par with the real-world hardware.
This is a best-case scenario that does not account for resource contention with other active virtual machines on the same physical server. It also does not account for certain vendor-specific I/O enhancement techniques, such as Direct I/O or Raw Device Mapping.
For recommendations on running Splunk Enterprise in a VMWare virtual machine, see "Deploying Splunk Enterprise Inside Virtual Environments" on the main Splunk website.
Splunk Enterprise, self-managed in the cloud
Running Splunk Enterprise in the cloud is a great alternative to running it on bare-metal hardware.
Given the hardware resources described earlier in this topic, Splunk Enterprise should deliver the same levels of performance on cloud infrastructure as it does on bare-metal hardware. That said, you should keep a close eye on what your cloud infrastructure actually delivers.
For example, if you run Splunk Enterprise on an Amazon Web Services (AWS) instance, note that:
- AWS measures CPU power on Elastic Compute Cloud (EC2) instances in virtual CPUs (vCPUs), not real CPUs.
- Each vCPU is a hyper thread of an Intel Xeon core on most AWS instance types. See "AWS | Amazon EC2 | Instance Types" (http://aws.amazon.com/ec2/instance-types/) on the AWS site.
- As a hyper thread of a core, a vCPU acts as a core, but the physical core must schedule its workload among other workloads of other vCPUs that the physical core handles.
For indexing and data storage, note that:
- If you choose to use Elastic Block Storage (EBS), the type of EBS volume you choose determines the amount of performance you get.
- Not all EBS volume types have the necessary IOPS to handle Splunk Enterprise operations.
- The "Provisioned IOPS" and "Magnetic" EBS volume types offer the best opportunity to get the required IOPS necessary for indexing and searching. See "EBS - Product Details" (http://aws.amazon.com/ebs/details/) on the AWS site.
- Not every EC2 instance type offers the network throughput to the EBS volume that you need. To ensure that bandwidth you must either launch the instance4 as "EBS-optimized" or choose an instance type that provides a minimum of 10Gb of bandwidth. See "Amazon EC2 Instance Configuration" (http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ebs-ec2-config.html) on the AWS site.
For forwarding, note that the proximity of your cloud infrastructure to your forwarders can have a major impact on performance of the whole environment.
Splunk offers its machine data platform and licensed software as a subscription service called Splunk Cloud. When you subscribe to the service, you purchase a capacity to index, store, and search your machine data. Splunk Cloud abstracts the infrastructure specification from you and delivers high performance on the capacity you have purchased.
Accommodate many simultaneous searches
This documentation applies to the following versions of Splunk® Enterprise: 6.2.0, 6.2.1, 6.2.2, 6.2.3, 6.2.4, 6.2.5, 6.2.6, 6.2.7, 6.2.8, 6.2.9, 6.2.10, 6.2.11, 6.2.12, 6.2.13, 6.2.14, 6.2.15