Performance test results and recommendations
Hardware performance test summary
Based on our test data, Splunk App for Stream uses approximately 14% of a single CPU core for each 100 Mbps TCP traffic processed, when using the default streams configuration (only the udp and tcp flow data streams are enabled). This increases to approximately 21% when application layer protocol processing is enabled for unencrypted traffic (by enabling the http stream), and approximately 29% when the traffic is encrypted using SSL.
The Splunk App for Stream test environment used:
- A single VM with four virtual cores running CentOS 6 64-bit on Vmware ESX host, with App for Stream version 6.3.0 running on Splunk version 6.3.
- Splunk_TA_stream (with
streamfwdbinary) installed on a universal forwarder, sending all data to a remote Splunk Indexer. ProcessingThreads is set to 4 in
- Neoload to simulate HTTP/S traffic.
- 2 web servers running Nginx to respond to HTTP/S traffic with a static 100 KB and 25KB HTML file.
- Every HTTP request is sent using a unique TCP connection (worst case scenario).
- Every HTTPS request is sent using a unique TCP connection and a unique SSL session (worst case scenario).
- For the SSL scenario (HTTPS), the private RSA is configured such that all requests are being decrypted.
These tables show hardware performance for 100KB and 25KB page sizes at increasing bandwidths.
|100KB page size||NON-SSL||SSL||TCP|
|Events per second (eps)||22||154||641||1354||2658||19||151||635||1324||9||82||336||722||1482|
|CPU % streamfwd (single core)||2||10||36||63||116||5||17||69||136||2||10||34||65||102|
|CPU % splunkd (single core)||1||3||11||25||57||1||3||11||23||1||2||6||14||29|
|Memory MB (streamfwd)||149||153||161||208||416||195||195||199||220||148||158||208||237||393|
|Memory MB (splunkd)||102||102||103||103||102||103||104||103||109||106||106||106||107||106|
|25KB page size||NON-SSL||SSL|
|Events per second (eps)||39||602||2540||75||597||2654|
|CPU % streamfwd (single core)||3||17||66||7||37||125|
|CPU % splunkd (single core)||1||5||23||1||5||20|
|Memory MB (streamfwd)||144||146||220||149||155||233|
|Memory MB (splunkd)||109||111||111||108||109||109|
This chart shows combined CPU usage (single core %) of
splunkd for 100KB page size.
This chart shows combined memory usage (MB) of
splunkd for 100KB page size.
Note: Hardware performance test results represent median values.
Stream forwarder sizing guide
The maximum number of stream forwarders (
streamfwd) that a Splunk App for Stream deployment can handle depends on the value of the
streamfwd.xml. Based on our tests, a single CPU core running Splunk App for Stream can process approximately 150 pings + stream configuration requests per second.
Use this formula to calculate the maximum number of stream forwarders your deployment can handle:
StreamForwarders = 150 * PingInterval * NumCores
PingInterval setting is 5 seconds, so a single CPU core should be able to handle up to 750 stream forwarders at peak activity (during startup up or when stream configuration changes are made).
If your deployment includes large numbers of stream forwarders, we recommend that you run Splunk App for Stream on a dedicated server AND stagger the startup of individual stream forwarders over time. Starting up too many stream forwarders simultaneously might overload the server.
Note: Based on our tests, during average usage, a single CPU core running Splunk App for Stream can process approximately 400 ping requests/second. However, the startup load on the CPU is significantly higher because
streamfwd must both send a ping request and pull down the full configuration from
Network collection architectures
This documentation applies to the following versions of Splunk Stream™: 6.3.0, 6.3.1, 6.3.2