Performance reference for the Azure Storage Blob input in the Splunk Add-on for Microsoft Cloud Services

The following is reference information about Splunk's performance testing of the Azure Storage Blob input in the Splunk Add-on for Microsoft Cloud Services. The testing occurred with version 4.1.4 of the Splunk Add-on for Microsoft Cloud Services. Use this information to enhance the performance of your own Azure Storage Blob data collection tasks.

Many factors impact performance results, including file size, file compression, event size, deployment architecture, and hardware. These results represent reference information and do not represent performance in all environments.

Blob input performance characteristics

Block blob

Common Architecture Setup	Blob Type	Event Size	Ingest Rate	Events per minute	Data lag	IDM CPU Cores % (Percentage of total IDM cores)
Scaled up stack 20 containers 20 add-on inputs 1 cluster master (c5.xlarge) 1 search head (c5.2xlarge) 3 indexer cluster (i3.large) 32 CPU cores 244 GB memory 1 input data manager (c5.9xlarge) 32 CPU cores 72 GB memory 20 Splunk Add-on for Microsoft Cloud Services inputs 120 seconds collection interval	block blob	10kb	48GB/day	3522	~120 seconds	1.8%
		1MB	50GB/day	36	~121 seconds	1.84%

Append blob

Common Architecture Setup	Blob Type	Event Size	Ingest Rate	Events per minute	Data lag	IDM CPU Cores % (Percentage of total IDM cores)
Scaled up stack 20 containers 20 add-on inputs 1 cluster master (c5.xlarge) 1 search head (c5.2xlarge) 3 indexer cluster (i3.large) 32 CPU cores 244 GB memory 1 input data manager (c5.9xlarge) 32 CPU cores 72 GB memory 20 Splunk Add-on for Microsoft Cloud Services inputs 120 seconds collection interval	append blob	1 kb	33GB/day	21522	~120 seconds	3.7%
		10 kb	281GB/day	20496	~121 seconds	2.67%
		100 kb	2.69TB/day	18402	~129 seconds	6.5%

Additional Blob input scenarios

Scenario	Workflow	Result
Append file up to 4GB	A single user creates an append blob. Append 100 kb each time. Append 40000 times. Pause up to 100 ms in each append.	All 40000 append blocks pulled back. Total event size matches original file size in Azure. Minimal to no data lag. Collection interval is 12 seconds.
Append 3 files sequentially to 100 MB	A single user creates an append blob. 100000 append each time. Pause up to 100 ms after each append. Append 1000 times to get to 100 MB for the file. Move on to next file and repeat. In total, append 3 files to 100 MB sequentially.	All 3000 append blocks pulled back. Total event size matches original file size in Azure.