Performance reference for the Azure Storage Blob input in the Splunk Add-on for Microsoft Cloud Services

The following tables show reference information about performance testing of the Azure Storage Blob input in the Splunk Add-on for Microsoft Cloud Services. Use this information to enhance the performance of your own Azure Storage Blob data collection tasks.

Many factors impact performance results, including file size, file compression, event size, deployment architecture, and hardware. These results represent reference information and do not represent performance in all environments.

Version 5.2.1 and above Blob input performance characteristics

Common architecture setup	Scenario	Blob type	File size	Ingest rate	IDM CPU
CO2 Stack - Victoria Search Head Cluster 1 cluster master (c5.xlarge) 3 search head (m6i.4xlarge) 16 CPU 64 GiB Memory 3 indexer cluster (im4gn.4xlarge) 16 CPU 64 GiB Memory Default TA configs
	1 input	AppendBlob	1MB	16324 kb/s	10.95%
	5 inputs	AppendBlob	1MB	39814 kb/s	27.69%
	1 input	BlockBlob	1MB	24364 kb/s	9.42%
	5 inputs	BlockBlob	1MB	40810 kb/s	16.34%
CO2 Stack - Classic Cluster (1 IDM) 1 cluster master (c5.xlarge) 3 search head (m6i.4xlarge) 16 CPU 64 GiB Memory 3 indexer cluster (im4gn.4xlarge) 16 CPU 64 GiB Memory 1 Inputs Data Manager (IDM) (m6i.4xlarge) 16 CPU 64 GiB Memory Default TA configs
	1 input	AppendBlob	1MB	8717 kb/s	8.44%
	5 inputs	AppendBlob	1MB	39815 kb/s	28.41%
	1 input	BlockBlob	1MB	11577 kb/s	6.4%
	5 inputs	BlockBlob	1MB	41857 kb/s	28%

Version 5.1.2 and above Blob input performance characteristics

Common architecture setup	Scenario	Blob type	File size	Ingest rate	IDM CPU
CO2 Stack - Victoria Search Head Cluster 1 cluster master (c5.xlarge) 3 search head (m6i.4xlarge) 16 CPU 64 GiB Memory 3 indexer cluster (im4gn.4xlarge) 16 CPU 64 GiB Memory Default TA configs
	1 input	AppendBlob	1MB	8277 kb/s	25.34%
	5 inputs	AppendBlob	1MB	37963 kb/s	28.09%
	1 input	BlockBlob	1MB	11532 kb/s	18.45%
	5 inputs	BlockBlob	1MB	40810 kb/s	29.82%
CO2 Stack - Classic Cluster (1 IDM) 1 cluster master (c5.xlarge) 3 search head (m6i.4xlarge) 16 CPU 64 GiB Memory 3 indexer cluster (im4gn.4xlarge) 16 CPU 64 GiB Memory 1 Inputs Data Manager (IDM) (m6i.4xlarge) 16 CPU 64 GiB Memory Default TA configs
	1 input	AppendBlob	1MB	7405 kb/s	9.45%
	5 inputs	AppendBlob	1MB	39815 kb/s	28.64%
	1 input	BlockBlob	1MB	20030 kb/s	10.12%
	5 inputs	BlockBlob	1MB	40810 kb/s	29.09%

Version 5.0.0 and above Blob input performance characteristics

Common architecture setup	Scenario	Blob type	File size	Ingest rate	IDM CPU
CO2 Stack - Victoria Search Head Cluster 1 cluster master (c5.xlarge) 3 search head (m6i.4xlarge) 16 CPU 64 GiB Memory 3 indexer cluster (is4gen.4xlarge) 16 CPU 96 GiB Memory Default TA configs
	1 input	AppendBlob	1MB	8155.26 kb/s	N/A
	5 inputs	AppendBlob	1MB	21390.76 kb/s	N/A
	1 input	BlockBlob	1MB	13100.21 kb/s	N/A
	5 inputs	BlockBlob	1MB	24328.95 kb/s	N/A
CO2 Stack - Classic Cluster (1 IDM) 1 cluster master (c5.xlarge) 3 search head (m6i.4xlarge) 16 CPU 64 GiB Memory 3 indexer cluster (is4gen.4xlarge) 16 CPU 96 GiB Memory 1 Inputs Data Manager (IDM) (m6i.4xlarge) 16 CPU 64 GiB Memory Default TA configs
	1 input	AppendBlob	1MB	6850.66 kb/s	55.85%
	5 inputs	AppendBlob	1MB	8049.20 kb/s	33.35%
	1 input	BlockBlob	1MB	3808.87 kb/s	44.55%
	5 inputs	BlockBlob	1MB	3838.31 kb/s	32.00%

Continuous append scenario
Append 6 files sequentially to 1GB.	A single user creates an append blob. Append 1MB each time in each file. Pause up to 1 second after each append. Append 1024 times to get to 1GB for the file. In total, append 6 files to 1GB in parallel.	Total event size matches the original file size in Azure.

Version 4.1.4 Blob input performance characteristics

Block blob

Common architecture setup	Blob type	File size	Ingest rate	Events per minute	Data lag	IDM CPU Cores % (Percentage of total IDM cores)
Scaled up stack 20 containers 20 add-on inputs 1 cluster master (c5.xlarge) 1 search head (c5.2xlarge) 3 indexer cluster (i3.large) 32 CPU cores 244 GB memory 1 input data manager (c5.9xlarge) 32 CPU cores 72 GB memory 20 Splunk Add-on for Microsoft Cloud Services inputs 120 seconds collection interval	block blob	10kb	48GB/day	3522	~120 seconds	1.8%
		1MB	50GB/day	36	~121 seconds	1.84%

Append blob

Common architecture setup	Blob type	File size	Ingest rate	Events per minute	Data lag	IDM CPU Cores % (Percentage of total IDM cores)
Scaled up stack 20 containers 20 add-on inputs 1 cluster master (c5.xlarge) 1 search head (c5.2xlarge) 3 indexer cluster (i3.large) 32 CPU cores 244 GB memory 1 input data manager (c5.9xlarge) 32 CPU cores 72 GB memory 20 Splunk Add-on for Microsoft Cloud Services inputs 120 seconds collection interval	append blob	1 kb	33GB/day	21522	~120 seconds	3.7%
		10 kb	281GB/day	20496	~121 seconds	2.67%
		100 kb	2.69TB/day	18402	~129 seconds	6.5%

Additional Blob input scenarios

Scenario	Workflow	Result
Append file up to 4GB	A single user creates an append blob. Append 100 kb each time. Append 40000 times. Pause up to 100 ms in each append.	All 40000 append blocks pulled back. Total event size matches original file size in Azure. Minimal to no data lag. Collection interval is 12 seconds.
Append 3 files sequentially to 100 MB	A single user creates an append blob. 100000 append each time. Pause up to 100 ms after each append. Append 1000 times to get to 100 MB for the file. Move on to the next file and repeat. In total, append 3 files to 100 MB sequentially.	All 3000 append blocks pulled back. Total event size matches original file size in Azure.