Splunk Stream

Installation and Configuration Manual

Download manual as PDF

This documentation does not apply to the most recent version of StreamApp. Click here for the latest version.
Download topic as PDF

Performance test results and recommendations

Hardware performance test summary

Based on our test data, Splunk App for Stream uses approximately 14% of a single CPU core for each 100 Mbps TCP traffic processed, when using the default streams configuration (only the udp and tcp flow data streams are enabled). This increases to approximately 21% when application layer protocol processing is enabled for unencrypted traffic (by enabling the http stream), and approximately 29% when the traffic is encrypted using SSL.

Test environment

The Splunk App for Stream test environment used:

  • A single VM with four virtual cores running CentOS 6 64-bit on Vmware ESX host, with App for Stream version 6.3.0 running on Splunk version 6.3.
  • Splunk_TA_stream (with streamfwd binary) installed on a universal forwarder, sending all data to a remote Splunk Indexer. ProcessingThreads is set to 4 in streamfwd.xml.
  • Neoload to simulate HTTP/S traffic.
  • 2 web servers running Nginx to respond to HTTP/S traffic with a static 100 KB and 25KB HTML file.
  • Every HTTP request is sent using a unique TCP connection (worst case scenario).
  • Every HTTPS request is sent using a unique TCP connection and a unique SSL session (worst case scenario).
  • For the SSL scenario (HTTPS), the private RSA is configured such that all requests are being decrypted.

Test results

These tables show hardware performance for 100KB and 25KB page sizes at increasing bandwidths.

100KB page size NON-SSL SSL TCP
Bandwidth (Mbps) 8 64 256 512 1000 8 64 256 512 8 64 256 512 1000
Events per second (eps) 22 154 641 1354 2658 19 151 635 1324 9 82 336 722 1482
CPU % streamfwd (single core) 2 10 36 63 116 5 17 69 136 2 10 34 65 102
CPU % splunkd (single core) 1 3 11 25 57 1 3 11 23 1 2 6 14 29
Memory MB (streamfwd) 149 153 161 208 416 195 195 199 220 148 158 208 237 393
Memory MB (splunkd) 102 102 103 103 102 103 104 103 109 106 106 106 107 106
25KB page size NON-SSL SSL
Bandwidth (Mbps) 8 64 256 8 64 256
Events per second (eps) 39 602 2540 75 597 2654
CPU % streamfwd (single core) 3 17 66 7 37 125
CPU % splunkd (single core) 1 5 23 1 5 20
Memory MB (streamfwd) 144 146 220 149 155 233
Memory MB (splunkd) 109 111 111 108 109 109

This chart shows combined CPU usage (single core %) of streamfwd and splunkd for 100KB page size.

Combined cpu usage 2.png

This chart shows combined memory usage (MB) of streamfwd and splunkd for 100KB page size.

Combined memory usage 2.png

Note: Hardware performance test results represent median values.

Stream forwarder sizing guide

The maximum number of stream forwarders (streamfwd) that a Splunk App for Stream deployment can handle depends on the value of the PingInterval in streamfwd.xml. Based on our tests, a single CPU core running Splunk App for Stream can process approximately 150 pings + stream configuration requests per second.

Use this formula to calculate the maximum number of stream forwarders your deployment can handle:

StreamForwarders = 150 * PingInterval * NumCores

The default PingInterval setting is 5 seconds, so a single CPU core should be able to handle up to 750 stream forwarders at peak activity (during startup up or when stream configuration changes are made).

If your deployment includes large numbers of stream forwarders, we recommend that you run Splunk App for Stream on a dedicated server AND stagger the startup of individual stream forwarders over time. Starting up too many stream forwarders simultaneously might overload the server.

Note: Based on our tests, during average usage, a single CPU core running Splunk App for Stream can process approximately 400 ping requests/second. However, the startup load on the CPU is significantly higher because streamfwd must both send a ping request and pull down the full configuration from splunk_app_stream.

Network collection architectures
Supported protocols

This documentation applies to the following versions of Splunk Stream: 6.3.0, 6.3.1, 6.3.2

Was this documentation topic helpful?

Enter your email address, and someone from the documentation team will respond to you:

Please provide your comments here. Ask a question or make a suggestion.

You must be logged into splunk.com in order to post comments. Log in now.

Please try to keep this discussion focused on the content covered in this documentation topic. If you have a more general question about Splunk functionality or are experiencing a difficulty with Splunk, consider posting a question to Splunkbase Answers.

0 out of 1000 Characters