Splunk® Supported Add-ons

Splunk Add-on for Amazon Kinesis Firehose

Performance reference for the Splunk Add-on for Amazon Kinesis Firehose

This page provides reference information on performance testing for version 1.1.1 of the Splunk Add-on for Amazon Kinesis Firehose. Use this information to enhance the performance of your own Amazon Kinesis Firehose instance.

Many factors impact performance results, including file size, file compression, event size, deployment architecture, and hardware. These results should be used as reference information and do not represent performance in all environments.

Testing architecture

The throughput data and conclusions provided here are based on performance testing using a paid Splunk Cloud deployment running in the following environment.

Instance type EC2 (c4.8xlarge)
Memory 60 GB
CPU 36 cores

Results

The following throughput data was measured with a single cluster master with seven indexers and one search head.

This table shows the average throughput for different sourcetypes achieved in performance testing under specific operating conditions and is subject to change when any of the hardware and software variables changes. These numbers are not indicative of Firehose performance but were measured using the backend HEC library that the Splunk Add-on for Amazon Kinesis Firehose uses. Use this data for a very rough reference only.

Sourcetype Throughput (MB/s)
Generic single line 142
CloudTrail 140
CloudWatch 1 138
CloudWatch 2 147
CloudWatch 3 110
CloudWatch 4 135
VPC Flow Logs 18

[1] We observed close to linear performance with the addition or subtraction of indexers. If you see traffic flowing to your Splunk instance mostly without exception with occasional "Could not connect to the HEC endpoint" errors, your HEC server may be too busy processing other data requests. Increase the number of HEC-enabled indexers to prevent your indexers from being filled. See the Accessing CloudWatch logs for Kinesis Firehose in the Monitoring with Amazon CloudWatch Logs documentation for instructions on how to view data delivery error logs.

Last modified on 08 October, 2021
Troubleshoot the Splunk Add-on for Amazon Kinesis Firehose  

This documentation applies to the following versions of Splunk® Supported Add-ons: released, released


Was this topic useful?







You must be logged into splunk.com in order to post comments. Log in now.

Please try to keep this discussion focused on the content covered in this documentation topic. If you have a more general question about Splunk functionality or are experiencing a difficulty with Splunk, consider posting a question to Splunkbase Answers.

0 out of 1000 Characters