Performance reference for the Splunk Add-on for Amazon Kinesis Firehose

This page provides reference information on performance testing for version 1.1.1 of the Splunk Add-on for Amazon Kinesis Firehose. Use this information to enhance the performance of your own Amazon Kinesis Firehose instance.

Many factors impact performance results, including file size, file compression, event size, deployment architecture, and hardware. These results should be used as reference information and do not represent performance in all environments.

Testing architecture

The throughput data and conclusions provided here are based on performance testing using a paid Splunk Cloud deployment running in the following environment.

Instance type	EC2 (c4.8xlarge)
Memory	60 GB
CPU	36 cores

Results

The following throughput data was measured with a single cluster master with seven indexers and one search head.

This table shows the average throughput for different sourcetypes achieved in performance testing under specific operating conditions and is subject to change when any of the hardware and software variables changes. These numbers are not indicative of Firehose performance but were measured using the backend HEC library that the Splunk Add-on for Amazon Kinesis Firehose uses. Use this data for a very rough reference only.

Sourcetype	Throughput (MB/s)
Generic single line	142
CloudTrail	140
CloudWatch 1	138
CloudWatch 2	147
CloudWatch 3	110
CloudWatch 4	135
VPC Flow Logs	18

[1] We observed close to linear performance with the addition or subtraction of indexers. If you see traffic flowing to your Splunk instance mostly without exception with occasional "Could not connect to the HEC endpoint" errors, your HEC server may be too busy processing other data requests. Increase the number of HEC-enabled indexers to prevent your indexers from being filled. See the Accessing CloudWatch logs for Kinesis Firehose in the Monitoring with Amazon CloudWatch Logs documentation for instructions on how to view data delivery error logs.