Performance reference for Splunk Enterprise Security

Review the following guidelines to optimize the performance of Splunk Enterprise Security prior to deploying the app on a configured Splunk platform installation:

• Considerations for scaling deployments
• Guidelines to optimize performance
• Constraints impacting performance
• Performance considerations for on-premises, cloud, and hybrid deployments
• Performance considerations for single instance and distributed search deployments
• Performance considerations when working with another Splunk product or app
• Performance considerations to deploy in virtualized environments

Performance test results

Review the following performance test results to estimate the performance you can expect from your infrastructure based on the mix of data in your Splunk platform and Enterprise Security deployment. The indexers used for these performance tests match the reference hardware with 32 GB of RAM and 16 CPU cores. Additionally, you must have a 64-bit operating system on all search heads and indexers to install Splunk Enterprise Security.

Performance criteria	Small deployment
Data ingestion	300 GB per day
Number of indexers	3
Number of correlation searches enabled	20

Considerations for scaling deployments

Evaluate your hardware, indexers, log size, and search heads to scale your Splunk Enterprise Security deployments.

Hardware scaling considerations

You might need to increase the hardware specifications of your Enterprise Security deployment beyond the minimum hardware requirements based on your environment. Depending on your system configuration, refer to the mid-range or high-performance specifications for Splunk platform reference hardware.

See Mid-range specification and High-performance specification in the Capacity Planning Manual.

Indexer scaling considerations

Indexing is an I/O-intensive process. The indexers require sufficient disk I/O to ingest and parse data efficiently while responding to search requests. For the latest IOPS requirements to run Splunk Enterprise, see Reference Hardware: Indexer in the Capacity Planning Manual.

To properly scale your distributed search deployment with Splunk Enterprise Security, see Indexer scaling considerations for Splunk Enterprise Security.

Increase the number of indexers in your deployment to scale with higher search load and search concurrency. Since a collection of indexers can serve more than one search head, additional search heads using the same indexers as a search head hosting Enterprise Security can affect the total performance of your indexer tier and reduce the resources available to Enterprise Security.

The Splunk platform uses indexers to scale horizontally. The number of indexers required in an Enterprise Security deployment varies based on the data volume, data type, retention requirements, search type, and search concurrency.

Work with Splunk Professional Services to estimate deployment architecture if you plan to ingest 1 terabyte (TB) per day or more of data into Enterprise Security. See Splunk Customer Success.

Log size scaling considerations

In a search head cluster environment, syncing large KV Store lookups across the cluster members can fail and cause the KV Store to become stale. To mitigate this, increase the operations log size. See Prevent stale members by increasing operations log size in the Splunk Enterprise Admin Manual.

Search head scaling considerations

You might need to increase the number of search heads based on the number of concurrent ad-hoc searches, real time searches, and enabled correlation searches. You might also need to increase the number of search heads based on the size of the asset and identity lookup files.

The following table provides information on scaling considerations for search heads when deploying Splunk Enterprise Security:

Factor	Increase this specification
A large number of concurrent ad hoc searches	Increase CPU cores and RAM
A large number of real-time searches being run or a large number of users logging in at the same time	Increase CPU cores
A large number of enabled correlation searches	Increase RAM
Large asset and identity lookup files	Increase RAM

The following tables provide guidance on how changing the data ingestion, data model acceleration, and search load might impact performance for Splunk Enterprise Security:

Deployment size	Data ingestion per day	Number of indexers	Number of correlation searches
Small	300 GB	3	20
Mid-range	1 TB	10	60
Mid-range to large	625 GB per day to 15 TB per day	24	60
Large	15 TB per day	150	100
Largest deployment tested in an on-premises search head cluster environment	45 TB with skip search rate of 4.9%	240	60
Largest deployment tested in on-premises single search head environment	25 TB with skip search rate of around 1%	300

Guidelines to optimize performance

Capacity planning is challenging due to the complexity of use cases, the data, and the architecture possibilities. Every situation is unique.

The following table highlights some best practices when planning to deploy Splunk Enterprise Security:

Capacity planning criteria	Installation best practice
Dedicated search head or search head cluster	Install Splunk Enterprise Security on a dedicated search head or a dedicated search head cluster.
Common Information Model add-on	Install only Common Information Model (CIM)-compatible apps or add-ons on the same search head as Splunk Enterprise Security. For example, you can install both the Splunk App for PCI Compliance and the Splunk Add-on Builder on the same search head as Splunk Enterprise Security.
Real time searches	All real-time searches in Splunk Enterprise Security use the indexed real-time setting to improve indexing performance. Deactivating the indexed real-time search setting reduces the overall indexing capacity of your indexers. To review the performance implications of the types of real-time searches, see Known limitations of real-time searches in the Search Manual.
Increase indexer capacity	Search head clusters increase the search load on indexers. Add more indexers or allocate more CPU cores to the indexers when implementing a search head cluster. See System requirements and other deployment considerations for search head clusters and Search head clustering architecture in the Splunk Enterprise Distributed Search Manual.

Constraints impacting performance

The following table describes the sizing constraints when deploying Splunk Enterprise Security:

Sizing criteria	Constraint
Correlation search load	Based on the number of correlation searches and supporting searches enabled in your deployment.
Data ingestion volume	Based on the volume of data being ingested into Splunk Enterprise Security
Data model acceleration load	Based on the number of data models being accelerated, the type of data being modeled, the cardinality of the data being modeled, and the volume of data being accelerated.
Indexer cluster support	Based on single-site or multi-site indexer clusters
Retention policy	Based on the index's time series index files (TSIDX)

Constraints on search load

Because high-volume Enterprise Security deployments run high numbers of searches that generate large amounts of results, the amount of work each peer must do can also exceed that of a smaller deployment. As a result, you must monitor and adjust memory consumption and run times of search jobs for safe levels.

Follow these best practices:

• Pay careful attention to the styles and types of searches that are allowed to run on high volume Enterprise Security deployments.
• Enforce quality standards against the types of SPL commands, timeframes, and intervals used for scheduled searches in Enterprise Security.

Constraints on data ingestion

When scaling Splunk Enterprise with Splunk Enterprise Security to data volumes exceeding 15 TB, some of the configurations that usually work in a Splunk Enterprise deployment will no longer work in a Splunk Enterprise deployment with Enterprise Security. The data model acceleration searches included with Enterprise Security impact overall cluster performance. Work with your Splunk field architect to calculate and validate large data volumes during deployment planning.

Constraints on data model acceleration

Depending on the data mix, the ingest volume, and the searches enabled, data model accelerations can lag behind the data ingestion.

Splunk Enterprise Security accelerates data models to provide dashboard, panel, and correlation search results. Data model acceleration uses the indexers for processing and storage, storing the accelerated data in each index.

Limit data model acceleration for specific data models to specific indexes to improve performance of data model acceleration and reduce indexer load, especially at scale. See Set up the Splunk Common Information Model Add-on for more on restricting data models to specific indexes.

See Data model acceleration storage and retention to calculate the additional storage for data model acceleration.

Constraints on retention policy for time series index

A retention policy for an index's time series index files (TSIDX) is available in Splunk Enterprise 6.4.x. For more information, see Reduce tsidx disk usage in the Splunk Enterprise Managing Indexers and Clusters of Indexers manual. Setting a retention policy for the TSIDX files does not affect the retention of data model accelerations.

Some searches provided with Enterprise Security do not work on buckets with reduced TSIDX files.

The following table provides guidelines on configuring the TSIDX retention value based on the panel or search name:

Panel or search name	Default time range	Workaround
Forwarder Audit panel: Event Count Over Time by Host	-30d	Set the TSIDX retention to a value greater than the time range.
Saved Search: Audit - Event Count Over Time By Top 10 Hosts	-30d	Set the TSIDX retention to a value greater than the time range.
Saved Search: Audit - Events Per Day - Lookup Gen	-1d	Set the TSIDX retention to a value greater than the default time range.
Saved Search: Endpoint - Index Time Delta 2 - Summary Gen	-1d	Set the TSIDX retention to a value greater than the default time range.

Constraints on type of indexer cluster

Splunk Enterprise Security supports both single site and multisite indexer cluster architectures. See The basics of indexer cluster architecture and Multisite indexer cluster architecture in Managing Indexers and Clusters of Indexers.

A single site or multisite indexer cluster architecture can have one search head or one search head cluster with a running instance of Enterprise Security. Additional single instance search heads or additional search head clusters cannot run Enterprise Security.

For a multisite indexer cluster architecture, follow these best practices:

• Enable summary replication. See Replicated summaries in Managing Indexers and Clusters of Indexers.
• Set the Enterprise Security search head to site0 to turn off search affinity. See Turn off search affinity in Managing Indexers and Clusters of Indexers.

If you use indexer clustering, the method you use to deploy apps and configuration files to indexer peers is different. See Manage common configurations across all peers and Manage app deployment across all peers in the Managing Indexers and Clusters of Indexers.

Performance considerations for on-premises, cloud, and hybrid deployments

You can deploy Splunk Enterprise Security on-premises and on Splunk Cloud Platform.

Deploying Splunk Enterprise Security on Splunk Cloud Platform

Review the following information to deploy Splunk Enterprise Security on Splunk Cloud Platform:

Splunk Enterprise Security is available as a service on the Splunk Cloud Platform. The Splunk Cloud Platform deployment architecture varies based on data and search load. Splunk Cloud Platform customers work with Splunk Support to set up, manage, and maintain their cloud infrastructure.

For information on Splunk Cloud Platform deployments, see the Splunk Cloud Platform deployment types in the Splunk Cloud Platform Admin Manual.

Deploying Splunk Enterprise Security in a hybrid environment

Review the following information to deploy Splunk Enterprise Security in a hybrid environment:

A hybrid search configuration with Splunk Enterprise Security is not supported with Splunk Cloud Platform. To set up a hybrid environment, set up an on-premises Splunk Enterprise Security search head to search indexers in another cloud environment. Any hybrid search deployment configuration must account for added latency, bandwidth concerns, and include adequate hardware to support the search load.

Performance considerations for single instance and distributed search deployments

Available deployment architectures to install Splunk Enterprise Security include a single instance deployment or a distributed search deployment.

Before you deploy Splunk Enterprise Security on premises, familiarize yourself with the components of a Splunk platform deployment. See Components of a Splunk Enterprise deployment in the Capacity Planning Manual.

Review the following performance considerations for single search head or a distributed search deployment before installing Enterprise Security:

Deployment type	Single-instance deployment	Distributed search deployment
Preferred	No. Usually used for a lab or test environment, or as a small system with one or two users running concurrent searches.	Yes
Search head requirements	A single platform instance functions as both a search head and indexer.	Install Splunk Enterprise Security on a dedicated search head or search head cluster
Indexer requirements	A single platform instance functions as both a search head and indexer.	To improve search performance, use an indexer cluster to distribute the search workload across multiple nodes. For a distributed search deployment, and for search head clustering, configure the search head to forward all data to the indexers. See Forward search head data to the indexer layer in the Distributed Search manual.
Data flow	Forwarders collect your data and send it to the single instance for parsing, storing, and searching.	Forwarders collect your data and send it to the indexers.
Supported operating system	Splunk Enterprise Security supports installation on Linux-based search head clusters only. Windows search head clusters are not supported. Additionally, stand-alone Windows servers cannot run Enterprise Security.	Splunk Enterprise Security supports installation on Linux-based search head clusters only. Windows search head clusters are not supported.

A dedicated search head might be required depending on the capacity of your specific environment and the workload of the apps you're already running and your Enterprise Security workload. See Introduction to capacity planning for Splunk Enterprise in the Splunk Enterprise Capacity Planning Manual.

Performance considerations when working with another Splunk product or app

Consider the following when using Splunk Enterprise Security with Splunk apps or add-ons:

• Managing apps or add-ons with the deployment server
• Support for app import and export
• Compatibility of Splunk Enterprise Security with other apps
• Mode of Monitoring Console

Managing apps or add-ons with the deployment server

Splunk Enterprise Security includes apps and add-ons. If the deployment server manages those apps or add-ons, Enterprise Security will not finish installing unless the following conditions are met:

• For add-ons included with Splunk Enterprise Security, deploy them using the Distributed Configuration Management tool. See Deploy add-ons to Splunk Enterprise Security.
• For other apps and add-ons installed in your environment, deploy them with the deployment server if appropriate. See About deployment server and forwarder management in Updating Splunk Enterprise Instances.
• For add-ons included with Splunk Enterprise Security and managed by a deployment server, remove the deployment client configuration before installing Enterprise Security. Remove the deploymentclient.conf file containing references to the deployment server and restart Splunk.

Support for app import and export

Splunk Enterprise Security does not selectively import apps and add-ons based on the name of the app or add-on. Knowledge objects in apps and add-ons that are installed on the same search head as Splunk Enterprise Security and exported to other apps or globally are visible in Splunk Enterprise Security.

To verify a global export from the search head, check the local.meta file of the app or add-on for export = system. For further details, see "Make Splunk knowledge objects globally available" in the Splunk Enterprise Admin Manual.

Compatibility of Splunk Enterprise Security with other apps

Splunk Enterprise Security relies on the search knowledge and CIM support supplied by add-ons. The add-ons are responsible for defining the event processing necessary to optimize, normalize, and categorize security data for use with the CIM. Only CIM-compatible apps are compatible with Splunk Enterprise Security. Other apps and add-ons that are not CIM-compatible can include data knowledge that is not normalized for the CIM, preventing searches and dashboards that rely on those fields from functioning properly.

Only install apps and add-ons on the same search head with Enterprise Security if they meet one of the following guidelines:

• Add-ons that are CIM-compatible and enrich data for use with Enterprise Security.
• Apps whose primary purpose is to integrate with Enterprise Security.

You can't install Splunk Enterprise Security and the SA-VMNetAppUtils component of the Splunk Add-on for VMware on the same search head. Conflicts with identically-named files can prevent some parts of Splunk Enterprise Security from working correctly.

Mode of Monitoring Console

If you enable the Monitoring Console on an Enterprise Security search head, it must remain in standalone mode. For more on when and how to configure the Monitoring Console in a distributed environment, see Which instance should host the console? in Monitoring Splunk Enterprise.

Performance considerations to deploy in virtualized environments

If you install Splunk Enterprise Security in a virtualized environment, you need the same memory and CPU allocation as a non-virtualized bare-metal environment.

Consider the following guidelines to deploy Splunk Enterprise Security in a virtualized environment:

• Reserve all CPU and memory resources.
• Do not oversubscribe hardware.
• Test the storage IOPS across all Splunk platform indexer nodes simultaneously to ensure that the IOPS match the reference hardware specification used in your environment. See Reference Hardware in the Capacity Planning Manual.

Insufficient storage performance is a common cause for poor search response and timeouts when scaling the Splunk platform in a virtualized environment.

• Use thick provisioned storage. Thin provisioning storage might impact performance.
• Hyper-threaded cores are not treated as extra cores. If you're running VMs on machines with hyper-threading enabled, you must double the vCPU count. For example, use 32 vCPUs instead of 16 physical cores.

See also

For more information on deployment planning, installation, and upgrading, see the product documentation:

• Install Splunk Enterprise Security

• System requirements for use of Splunk Enterprise on-premises in the Splunk Enterprise Installation Manual.

• Splunk Products Version Compatibility Matrix in the Splunk Products Version Compatibility Matrix manual.

• Splunk Cloud Platform Service Details in the Splunk Cloud Platform Service Description manual.

• About upgrading to 8.0 READ THIS FIRST in the Splunk Enterprise Installation Manual.