Optimize Splunk Enterprise for peak performance

This topic discusses standards that assist the system administrator when implementing or expanding their Splunk Enterprise infrastructure, and in maintaining consistent performance:

• Designate one or more machines solely for Splunk Enterprise components. Splunk scales horizontally. Adding more physical machines dedicated to Splunk Enterprise translates into better performance than having more resources in a single machine. Where possible, split up your indexing and searching activities across a number of machines, and only run one Splunk Enterprise component on each machine. Performance is reduced when you run Splunk Enterprise on machines that share resources with other services.

• Provide dedicated, fast storage to your Splunk Enterprise indexers. Insufficient storage I/O is the most commonly encountered limitation in a Splunk software infrastructure. For guidance on storage for indexers, see What storage type should I use for a role? in the Capacity Planning Manual.

• Don't allow anti-virus programs to scan disks used for Splunk services. When an anti-virus product scans files for viruses on access, performance of Splunk services is significantly reduced, especially as the recently indexed data ages. If you use anti-virus programs on the servers running Splunk Enterprise, make sure that all Splunk software directories and programs are excluded from on-access file scans.

• Use multiple indexes, where possible. Distribute the data that in indexed by Splunk into different indexes. Sending all data to one index can cause I/O bottlenecks on your system and complicate retention calculations and access controls. For information on how to configure indexes, see Configure your indexes in the Managing Indexers and Clusters of Indexers manual.

• Don't store your indexes on the same physical disk or volume as the operating system. The disk that holds your operating system or its swap file is not a recommended place for Splunk Enterprise data storage. Put your indexes on other disks or volumes mounted on the machine. For more information on how indexes are stored, including information on database bucket types and how Splunk stores and ages them, see How Splunk stores indexes in the Managing Indexers and Clusters of Indexers manual.

• Don't store the hot and warm buckets of your indexes on network volumes. Network latency will decrease indexing performance significantly. Always use fast, local disk for the index hot and warm buckets. You can specify network shares for the cold and frozen buckets of an index using Distributed File System (DFS) volumes or Network File System (NFS) mounts. But searches that include data stored on network volumes will be slower.

• Maintain disk availability, bandwidth, and space on your indexers. Make sure that the disk volumes or mounts that hold the indexes maintain free space at all times. Disk performance decreases as available space decreases, and disk seek times will increase. Slow storage affects how efficiently Splunk Enterprise indexes data, and will also impact how quickly search results, reports and alerts are returned. The volume or mount that contains your indexes must have approximately 5 gigabytes of free disk space by default, or indexing will stop.