Splunk® Supported Add-ons

Splunk Add-on for CrowdStrike FDR

Acrobat logo Download manual as PDF


Acrobat logo Download topic as PDF

Performance reference for the Splunk Add-on for CrowdStrike

This page provides reference information about Splunk's performance testing for the Splunk Add-on for CrowdStrike.

Performance results should be used as reference information and do not represent performance in all environments. Many factors impact performance results, including:

  • file size
  • file compression
  • event size
  • deployment architecture
  • hardware

When preparing instances to ingest CrowdStrike data, consider Compute Optimized Instances, as ingestion is mostly CPU intensive and prefers higher clock CPUs. For each pipeline reserve 4-6 cores. Ingesting bigger volumes of CrowdStrike data also requires more indexer resources, one ingestor pipeline is sufficient to feed up to three indexing pipelines.

To increase the number of ingest pipelines in Splunk Cloud Victoria, contact the Splunk Cloud support team to request an exception.

Hardware and software environment

The throughput data and conclusions provided in this topic are based on performance testing using the Add ModInput functionality.

Instance type C5 Double Extra Large (c5.4xlarge)
Memory 32 GB
Compute Units (ECU) 73
vCPU 16

Measured performance data using HF/IDM

The data throughput is based on the S3 key size along with events and number of configured SQS-Based S3 inputs configured on heavy forwarders and input data managers.

Data input Compressed Batch size (GB) HF/IDM number inputs per HF/IDM Managed S3 consumers Indexer Batch processing time [min] Max throughput (KBs) Max throughput (GB/day)
SQS-based S3 2.2 1 1 - 3 20 50,000 4,100
SQS based manager 2.2 1 1 1 3 37 28,000 2,300
SQS based manager 2.2 1 1 2 3 20 51,000 4,200
SQS based manager 2.2 1 1 4 3 11 92,000 7,600
SQS based manager 2.2 1 1 8 3 8 134,500 11,000

Configure parallelism on data collection nodes for SQS-based S3 input

To ensure that your solution scales properly, set the parallelIngestionPipelines number to match the number of SQS-based S3 inputs per HF/IDM/SH. For example, if each HF/IDM/SH has four inputs, then you set parallelIngestionPipelines = 4 in $SPLUNK_HOME/etc/system/local/server.conf.

Adding additional inputs will not change the performance if the parameter is set lower. By default this parameter is set to 1, and you should not add more than a single SQS-based S3 input per heavy forwarder or IDM with the default parallelism configuration. Each instance is managed separately, so check your desired configuration on all data collection nodes.

For Splunk Cloud Platform Victoria, search heads serve as heavy forwarders. However, input replication for Splunk Cloud version 8.2.2201 and later uses a different input replication process. CrowdStrike FDR SQS-based S3 consumer inputs are configured globally and replicated for each member in the search-head cluster. You do not need to manage each search head separately. To add more than a single input, contact Cloud Support to increase parallelIngestionPipelines on search heads.

To increase the number of ingest pipelines in Splunk Cloud Victoria, contact the Splunk Cloud support team to request an exception.

When a single SQS message consists of multiple files, a single mod input processes them sequentially where one input is in use even if others are idle. In this case, use instances with better clock speed. CrowdStrike creates a new event batch every 7-10 minutes. In heavy loaded environments, event batches with hundreds of files is common. A batch containing for example 300-400 files can be processed for 4-7 hours, meaning that one input will be busy during this time and not capable of ingesting the next generated batch. Make sure you have enough inputs to start processing the next batch as it appears.

Configure parallelism on data collection nodes for SQS based manager input

With SQS based manager input, parallelism is achieved by adding more managed consumer inputs. For any parallelIngestionPipelines configuration on the Ingestion Node, you can run a single SQS based manager, but you can configure this with multiple consumers. You should configure at least two consumers and to add more to match the number of configured parallelIngestionPipelines.

Scaling forwarders horizontally

Add more forwarders to safely increase the throughput in a heavy-load environment. To achieve daily throughput above 10TB per day, configure two heavy forwarders with four inputs on each instance. Processing on the indexers could create a bottleneck; forwarding data to at least six indexers will achieve better performance. If adding the new Ingestion node does not increase the whole stack throughput or other nodes have lower throughput per instance, this most likely means there are not enough indexers in the stack.

Cloud Stack recommendations

For Enterprise Cloud Platform search head instances, CPU processing is shared between search and ingestion and more resources are reserved for search. To achieve throughput of 10TB/day without impacting search experience, create at least six search heads (c5.4xlarge) and six indexers (i3.8xlarge). When using a Splunk Classic stack with a single IDM, use at least an c6i.12xlarge instance with 8 inputs and parallelisation to achieve similar throughput.

Enabling index time host resolution

Depending on data, host resolution on index time can slow data ingestion by 5-35%. Performing host resolution at index time consumes CPU on the forwarder. Monitor your data ingestion before turning on this feature, and configure Crowdstrike FDR host information sync if your ingestor node reaches CPU limits. In case of any delays during ingestion, consider upgrading your setup or turning off this feature.

Last modified on 14 December, 2023
PREVIOUS
Scripted bitmask lookups for the Splunk Add-on for Crowdstrike
  NEXT
Crowdstrike FDR data volumes

This documentation applies to the following versions of Splunk® Supported Add-ons: released


Was this documentation topic helpful?


You must be logged into splunk.com in order to post comments. Log in now.

Please try to keep this discussion focused on the content covered in this documentation topic. If you have a more general question about Splunk functionality or are experiencing a difficulty with Splunk, consider posting a question to Splunkbase Answers.

0 out of 1000 Characters