Performance reference for the Splunk Add-on for CrowdStrike
This page provides reference information about Splunk's performance testing for the Splunk Add-on for CrowdStrike.
Performance results should be used as reference information and do not represent performance in all environments. Many factors impact performance results, including:
- file size
- file compression
- event size
- deployment architecture
When preparing instances to ingest CrowdStrike data, consider Compute Optimized Instances, as ingestion is mostly CPU intensive and prefers higher clock CPUs. For each input/pipeline reserve 4-6 cores.
Hardware and software environment
The throughput data and conclusions provided in this topic are based on performance testing using the Add ModInput functionality.
|Instance type||M4 Double Extra Large (m4.4xlarge)|
|Compute Units (ECU)||53.5|
Measured performance data using HF/IDM
The data throughput is based on the S3 key size along with events and number of configured SQS-Based S3 inputs configured on heavy forwarders and input data managers.
|S3 key Size (MB)||HF/IDM number||inputs per HF/IDM||Indexers||Max throughput (KBs)||Max EPS (events)||Max throughput (GB/day)|
Configure parallelism on data collection nodes
To ensure that the solution scales properly, set the
parallelIngestionPipelines number to match the number of inputs per HF/IDM/SH. For example, if each HF/IDM/SH has four inputs, then you set parallelIngestionPipelines = 4 in
Adding additional inputs will not change the performance if the parameter is set lower. By default this parameter is set to 1, and you should not add more than a single input per heavy forwarder or IDM with the default parallelism configuration. Each instance is managed separately, so check your desired configuration on all data collection nodes.
For Splunk Cloud Platform Victoria, search heads serve as heavy forwarders. However, input replication for Splunk Cloud version 8.2.2201 and later uses a different input replication process. CrowdStrike FDR SQS-based S3 consumer inputs are configured globally and replicated for each member in the search-head cluster. You do not need to manage each search head separately. To add more than a single input, contact Cloud Support to increase
parallelIngestionPipelines on search heads.
When a single SQS message consists of multiple files, a single mod input processes them sequentially where one input is in use even if others are idle. In this case, use instances with better clock speed. CrowdStrike creates a new event batch every 7-10 minutes. In heavy loaded environments, event batches with hundreds of files is common. A batch containing for example 300-400 files can be processed for 4-7 hours, meaning that one input will be busy during this time and not capable of ingesting the next generated batch. Make sure you have enough inputs to start processing the next batch as it appears.
Scaling forwarders horizontally
Add more forwarders to safely increase the throughput in a heavy-load environment. To achieve daily throughput above 10TB per day, configure four heavy forwarders with four inputs on each instance. Processing on the indexers could create a bottleneck; forwarding data to at least four indexers will achieve better performance.
Cloud Stack recommendations
For Enterprise Cloud Platform search head instances, CPU processing is shared between search and ingestion and more resources are reserved for search. To achieve throughput of 10TB/day without impacting search experience, create at least six search heads (c5.4xlarge) and six indexers (i3.8xlarge). When using a Splunk Classic stack with a single IDM, use at least an c6i.12xlarge instance with 8 inputs and parallelisation to achieve similar throughput.
Enabling index time host resolution
Depending on data, host resolution on index time can slow data ingestion by 5-35%. Monitor your data ingestion when turning on this feature. In case of any delays during ingestion, consider upgrading your setup or turning off this feature.
Scripted bitmask lookups for the Splunk Add-on for Crowdstrike
Crowdstrike FDR data volumes
This documentation applies to the following versions of Splunk® Supported Add-ons: released