Performance test results and recommendations

Hardware performance test summary

Based on our test data, Splunk App for Stream uses approximately 14% of a single CPU core for each 100 Mbps TCP traffic processed, when using the default streams configuration (only the udp and tcp flow data streams are enabled). This increases to approximately 21% when application layer protocol processing is enabled for unencrypted traffic (by enabling the http stream), and approximately 29% when the traffic is encrypted using SSL.

Test environment

The Splunk App for Stream test environment used:

• A single VM with four virtual cores running CentOS 6 64-bit on Vmware ESX host, with App for Stream version 6.3.0 running on Splunk version 6.3.

• Splunk_TA_stream (with streamfwd binary) installed on a universal forwarder, sending all data to a remote Splunk Indexer. ProcessingThreads is set to 4 in streamfwd.xml.

• Neoload to simulate HTTP/S traffic.

• 2 web servers running Nginx to respond to HTTP/S traffic with a static 100 KB and 25KB HTML file.

• Every HTTP request is sent using a unique TCP connection (worst case scenario).

• Every HTTPS request is sent using a unique TCP connection and a unique SSL session (worst case scenario).

• For the SSL scenario (HTTPS), the private RSA is configured such that all requests are being decrypted.

Test results

These tables show hardware performance for 100KB and 25KB page sizes at increasing bandwidths.

100KB page size	NON-SSL					SSL				TCP
Bandwidth (Mbps)	8	64	256	512	1000	8	64	256	512	8	64	256	512	1000
Events per second (eps)	22	154	641	1354	2658	19	151	635	1324	9	82	336	722	1482
CPU % streamfwd (single core)	2	10	36	63	116	5	17	69	136	2	10	34	65	102
CPU % splunkd (single core)	1	3	11	25	57	1	3	11	23	1	2	6	14	29
Memory MB (streamfwd)	149	153	161	208	416	195	195	199	220	148	158	208	237	393
Memory MB (splunkd)	102	102	103	103	102	103	104	103	109	106	106	106	107	106

25KB page size	NON-SSL			SSL
Bandwidth (Mbps)	8	64	256	8	64	256
Events per second (eps)	39	602	2540	75	597	2654
CPU % streamfwd (single core)	3	17	66	7	37	125
CPU % splunkd (single core)	1	5	23	1	5	20
Memory MB (streamfwd)	144	146	220	149	155	233
Memory MB (splunkd)	109	111	111	108	109	109

This chart shows combined CPU usage (single core %) of streamfwd and splunkd for 100KB page size.

This chart shows combined memory usage (MB) of streamfwd and splunkd for 100KB page size.

Note: Hardware performance test results represent median values.

Stream forwarder sizing guide

The maximum number of stream forwarders (streamfwd) that a search head can support depends on the value of the PingInterval in streamfwd.xml. Based on our tests, a single search running Splunk App for Stream can process approximately 150 pings + stream configuration requests per second.

Use this formula to calculate the maximum number of stream forwarders per search head:

StreamForwarders per search head = 150 * PingInterval

The default PingInterval setting is 5 seconds, so a single search head can run up to 750 stream forwarders at peak activity (during startup up or when stream configuration changes are made).

If your deployment includes large numbers of stream forwarders, we recommend that you run Splunk App for Stream on a dedicated server AND stagger the startup of individual stream forwarders over time. Starting up too many stream forwarders simultaneously might overload the server.

Note: Based on our tests, during average usage, a single search head running Splunk App for Stream can process approximately 400 ping requests/second. However, the startup load on the CPU is significantly higher because streamfwd must both send a ping request and pull down the full configuration from splunk_app_stream.