Splunk® Supported Add-ons

Splunk Add-on for CrowdStrike FDR

Acrobat logo Download manual as PDF


Acrobat logo Download topic as PDF

Performance reference for the Splunk Add-on for CrowdStrike

This page provides reference information about Splunk's performance testing for the Splunk Add-on for CrowdStrike.

Performance results should be used as reference information and do not represent performance in all environments. Many factors impact performance results, including:

  • file size
  • file compression
  • event size
  • deployment architecture
  • hardware

When preparing instances to ingest CrowdStrike data, consider Compute Optimized Instances, as ingestion is mostly CPU intensive and prefers higher clock CPUs. For each input/pipeline reserve 4-6 cores.

Hardware and software environment

The throughput data and conclusions provided in this topic are based on performance testing using the Add ModInput functionality.

Instance type M4 Double Extra Large (m4.4xlarge)
Memory 64 GB
Compute Units (ECU) 53.5
vCPU 16

Measured performance data using HF/IDM

The data throughput is based on the S3 key size along with events and number of configured SQS-Based S3 inputs configured on heavy forwarders and input data managers.

S3 key Size (MB) HF/IDM number inputs per HF/IDM Indexers Max throughput (KBs) Max EPS (events) Max throughput (GB/day)
25 1 1 8 18,000 20,000 1,500
25 1 2 8 34,000 37,000 2,800
25 1 4 8 44,000 48,000 3,600
25 1 8 8 62,000 68,000 5,100
25 4 4 8 185,000 204,000 15,300
25 4 8 8 244,000 268,000 20,100

Configure parallelism on data collection nodes

To ensure that the solution scales properly, set the parallelIngestionPipelines number to match the number of inputs per HF/IDM/SH. For example, if each HF/IDM/SH has four inputs, then you set parallelIngestionPipelines = 4 in $SPLUNK_HOME/etc/system/local/server.conf.

Adding additional inputs will not change the performance if the parameter is set lower. By default this parameter is set to 1, and you should not add more than a single input per heavy forwarder or IDM with the default parallelism configuration. Each instance is managed separately, so check your desired configuration on all data collection nodes.

For Splunk Cloud Platform Victoria, search heads serve as heavy forwarders. However, input replication for Splunk Cloud version 8.2.2201 and later uses a different input replication process. CrowdStrike FDR SQS-based S3 consumer inputs are configured globally and replicated for each member in the search-head cluster. You do not need to manage each search head separately. To add more than a single input, contact Cloud Support to increase parallelIngestionPipelines on search heads.

When a single SQS message consists of multiple files, a single mod input processes them sequentially where one input is in use even if others are idle. In this case, use instances with better clock speed. CrowdStrike creates a new event batch every 7-10 minutes. In heavy loaded environments, event batches with hundreds of files is common. A batch containing for example 300-400 files can be processed for 4-7 hours, meaning that one input will be busy during this time and not capable of ingesting the next generated batch. Make sure you have enough inputs to start processing the next batch as it appears.

Scaling forwarders horizontally

Add more forwarders to safely increase the throughput in a heavy-load environment. To achieve daily throughput above 10TB per day, configure four heavy forwarders with four inputs on each instance. Processing on the indexers could create a bottleneck; forwarding data to at least four indexers will achieve better performance.

Cloud Stack recommendations

For Enterprise Cloud Platform search head instances, CPU processing is shared between search and ingestion and more resources are reserved for search. To achieve throughput of 10TB/day without impacting search experience, create at least six search heads (c5.4xlarge) and six indexers (i3.8xlarge). When using a Splunk Classic stack with a single IDM, use at least an c6i.12xlarge instance with 8 inputs and parallelisation to achieve similar throughput.

Enabling index time host resolution

Depending on data, host resolution on index time can slow data ingestion by 5-35%. Monitor your data ingestion when turning on this feature. In case of any delays during ingestion, consider upgrading your setup or turning off this feature.

Last modified on 26 April, 2022
PREVIOUS
Scripted bitmask lookups for the Splunk Add-on for Crowdstrike
  NEXT
Crowdstrike FDR data volumes

This documentation applies to the following versions of Splunk® Supported Add-ons: released


Was this documentation topic helpful?


You must be logged into splunk.com in order to post comments. Log in now.

Please try to keep this discussion focused on the content covered in this documentation topic. If you have a more general question about Splunk functionality or are experiencing a difficulty with Splunk, consider posting a question to Splunkbase Answers.

0 out of 1000 Characters