Performance reference for the Splunk Add-on for Google Cloud Platform

This page provides reference information on performance testing of the pub/sub, pub/sub lite, and pub/sub based bucket inputs for the Splunk Add-on for Google Cloud Platform. Use this information to enhance the performance of your own Google Cloud Platform collection tasks.

Many factors impact performance results, including file size, file compression, event size, deployment architecture, and hardware. These results should be used as reference information and do not represent performance in all environments.

Version 4.5.0 Cloud Storage Bucket input performance statistics for JSON files

Common architecture setup	Number of inputs	Number of Threads	Number of files	File Size	Events Count	Data Collection Time (in minutes)	Avg CPU (by input)	Avg Memory (by input)
CO2 Stack - Victoria Search Head Cluster 3 search heads (m5.4xlarge) 16 CPU 64 GiB memory 3 indexers (im4gn.large) 1 cluster master (c5.xlarge)	1 input	1	10k	~3MB	6180001	53	3%	0.1%
	1 input	10	10k	~3MB	6180001	12	15%	0.1%
CO2 Stack - Classic Cluster (1 IDM) 1 cluster master (c5.xlarge) 3 search heads (c5.2xlarge) 3 indexers (i3.large) 1 Input Data Manager (IDM) (m5.4xlarge) 16 CPU 64 GiB memory	1 input	1	10k	~3MB	6180001	48	4%	0.1%
	1 input	10	10k	~3MB	6180001	15	13%	0.1%
On Prem Heavy Forwarder (m5.4xlarge) 16 CPU 64 GiB memory	1 input	1	10k	~3MB	6180001	108	2%	0.1%
	1 input	10	10k	~3MB	6180001	14	12%	0.1%

Version 4.5.0 Cloud Storage Bucket input performance statistics for CSV files

Common architecture setup	Number of inputs	Number of Threads	Number of files	File Size	Events Count	Data Collection Time (in minutes)	Avg CPU (by input)	Avg Memory (by input)
CO2 Stack - Victoria Search Head Cluster 3 search heads (m5.4xlarge) 16 CPU 64 GiB memory 3 indexers (im4gn.large) 1 cluster master (c5.xlarge)	1 input	1	100k	~50KB	86159944	225	6%	0.1%
	1 input	10	100k	~50KB	86159944	169	15%	0.1%
CO2 Stack - Classic Cluster (1 IDM) 1 cluster master (c5.xlarge) 3 search heads (c5.2xlarge) 3 indexers (i3.large) 1 Input Data Manager (IDM) (m5.4xlarge) 16 CPU 64 GiB memory	1 input	1	100k	~50KB	86159944	223	5%	0.1%
	1 input	10	100k	~50KB	86159944	167	15%	0.1%
On Prem Heavy Forwarder (m5.4xlarge) 16 CPU 64 GiB memory	1 input	1	100k	~50KB	86159944	445	3%	0.1%
	1 input	10	100k	~50KB	86159944	155	15%	0.1%

Version 4.2.0 Pub/Sub Lite input performance statistics

There following tests a configuration of 20 partitions for Version 4.2.0 Pub/Sub Lite input performance.

Common architecture setup	Scenario	Event type	Event Size	Number of threads	Ingest Rate (KB/min)	CPU	RAM
CO2 Stack - Victoria Search Head Cluster 1 cluster master (c5.xlarge) 3 search heads (c6i.4xlarge) 16 CPU 16 GiB Memory 3 indexer clusters (is4gen.4xlarge) 16 CPU core 96 GiB Memory Default TA configs	1 input	JSON	1 KB	10	212274	13.58%	11.8%
	1 input	JSON	1 KB	64	2202370	14.25%	11.7%
	20 input	JSON	1 KB	10	2020845	22.88%	9.74%
	20 input	JSON	1 KB	64	2067952	23.25%	8.79%
	1 input	JSON	10KB	10	3166630	14.32%	6.77%
	20 input	JSON	10KB	10	6197910	22.05%	11.07%
	1 input	Non-JSON	1KB	10	414976	13.69%	8.05%
	1 input	Non-JSON	1KB	64	379032	13.78%	3.27%
	20 input	Non-JSON	1KB	10	2069770	23.9%	8.63%
	20 input	Non-JSON	1KB	64	2064460	23.2%	7.06%
	1 input	Non-JSON	10KB	10	3795480	14.48%	8.92%
	20 input	Non-JSON	10KB	10	6129498	23.7%	5.57%
CO2 Stack - Classic Cluster (1 IDM) 1 cluster master (c5.xlarge) 3 search heads (c6i.xlarge) 4 CPU core 8 GB RAM 3 indexers (is4gen.4xlarge) 16 CPU 96 GiB Memory 1 Input Data Manager (IDM) (c6i.4xlarge) 16 CPU core 32 GB RAM Default TA configs
	1 input	JSON	1KB	10	231987	13.61%	6.6%
	1 input	JSON	1KB	64	237228	13.65%	6.65%
	20 input	JSON	1KB	10	1022357	22.4%	13.1%
	20 input	JSON	1KB	64	1019737	22.7%	12.98%
	1 input	JSON	10KB	10	1887750	14.21%	12.9%
	20 input	JSON	10KB	10	3228910	23.38%	13.1%
	1 input	Non-JSON	1KB	10	247171	13.6%	12.96%
	1 input	Non-JSON	1KB	64	241652	9.17%	13.55%
	20 input	Non-JSON	1KB	10	1021291	22.5%	3.9%
	20 input	Non-JSON	1KB	64	987298	23%	4.02%
	1 input	Non-JSON	10KB	10	2171011	14.35%	4%
	20 input	Non-JSON	10KB	10	291004	22%	4.05%

Version 4.2.0 Pub/Sub Based Bucket input performance statistics

The following tests file sizes of 1, 5, and 10 KB for Version 4.2.0 Pub/Sub Based Bucket input. The number of threads tested is 10.

Common architecture setup	Scenario	File type	Message batch	Ingest rate (KB/min)	CPU	RAM
CO2 Stack - Victoria Search Head Cluster 1 cluster master (c5.xlarge) 3 search head (c6i.4xlarge) 16 CPU 32 GiB Memory 3 indexer cluster (is4gen.4xlarge) 16 CPU 96 GiB Memory Default TA configs
	1 input	CSV	10	1649	3.75%	11.55%
	5 inputs	CSV	10	7869	3.7%	11.54%
	10 inputs	CSV	10	16670	4.65%	11.53%
	20 inputs	CSV	10	33327	3.75%	11.49%
	40 inputs	CSV	10	64360	4.3%	11.48%
	1 input	CSV	100	9783	3.9%	11.56%
	5 inputs	CSV	100	49756	3.65%	11.54%
	10 inputs	CSV	100	100888	3.85%	11.54%
	20 inputs	CSV	100	192448	4.49%	11.52%
	40 inputs	CSV	100	385078	7.46%	11.54%
	1 input	JSON	10	1655	4.25%	11.59%
	5 inputs	JSON	10	7853	4.5%	11.58%
	10 inputs	JSON	10	16882	3.65%	11.59%
	20 inputs	JSON	10	34611	4.65%	11.57%
	40 inputs	JSON	10	65909	3.9%	11.57%
	1 input	JSON	100	9813	3.85%	11.57%
	5 inputs	JSON	100	48865	3.55%	11.55%
	10 inputs	JSON	100	99252	4.4%	11.52%
	20 inputs	JSON	100	201040	4.24%	11.46%
	40 inputs	JSON	100	397136	7.89%	11.39%
CO2 Stack - Classic Cluster (1 IDM) 1 cluster master (c5.xlarge) 3 search heads (c6i.xlarge) 4 CPU core 8 GB RAM 3 indexers (is4gen.4xlarge) 16 CPU 96 GiB Memory 1 Input Data Manager (IDM) (c6i.4xlarge) 16 CPU core 32 GB RAM Default TA configs	1 input	CSV	10	1596	4.5%	9.87%
	5 inputs	CSV	10	7670	4.25%	9.84%
	10 inputs	CSV	10	15129	4.75%	9.48%
	20 inputs	CSV	10	30278	4.45%	9.06%
	40 inputs	CSV	10	60220	6.5%	8.68%
	1 input	CSV	100	8708	5.3%	9.89%
	5 inputs	CSV	100	45657	4.45%	9.46%
	10 inputs	CSV	100	92601	4.91%	9.15%
	20 inputs	CSV	100	185263	6.57%	8.86%
	40 inputs	CSV	100	365956	9.09%	8.45%
	1 input	JSON	10	1591	5.7%	6.98%
	5 inputs	JSON	10	7550	6.95%	4.4%
	10 inputs	JSON	10	15376	4.75%	6.88%
	20 inputs	JSON	10	30534	5.5%	6.7%
	40 inputs	JSON	10	60647	8.05%	6.23%
	1 input	JSON	100	11024	4.3%	8.43%
	5 inputs	JSON	100	48298	4%	7.97%
	10 inputs	JSON	100	93452	4.81%	7.53%
	20 inputs	JSON	100	182911	6.92%	7.44%
	40 inputs	JSON	100	372861	6.78%	7.54%

Version 1.3.0 Pub/Sub input performance statistics

Use the following information on version 1.3.0 Pub/Sub input performance statistics to enhance the performance of your own Google Cloud Platform collection tasks.

Testing architecture

The throughput data and conclusions provided here are based on performance testing using single-instance Splunk Enterprise 7.0.1 running on the following environment.

Instance type	n1-standard-8
Memory	30 GB
vCPU	8
Cores	4 CPU cores
Storage Type	standard persistent disk (2000GB)

Measured performance data

The throughput data provided here is the average performance for different subscription numbers achieved in performance testing under specific operating conditions and is subject to change when any of the hardware and software variables changes. Use this data for a rough reference only.

Subscriptions	Input numbers	Throughput (KB/s)	Throughput (GB/day)
1	1	1800	150
4	4	9000	740
8	8	17000	1400
16	16	24000	1970
200	16	16320	1340

For heavy data ingestion scenarios, change the default Acknowledgment Deadline of your subscription from 10 seconds to 60 seconds for optimal performance and to avoid data duplication.