Splunk Stream

Installation and Configuration Manual

Download manual as PDF

This documentation does not apply to the most recent version of StreamApp. Click here for the latest version.
Download topic as PDF

Splunk Stream test environments

This page describes the various test environments used in Splunk Stream hardware performance tests.

Splunk Stream performance test results show CPU usage and Memory usage of splunkd and streamfwd for HTTP and TCP/UDP traffic over a range of workloads, both with and without SSL. Hardware performance tests are run on the following Splunk Stream features:

  • Splunk_TA_stream (which contains thestreamfwd binary) running on a Universal forwarder (UF).
  • Independent Stream forwarder (streamfwd binary) sending data to indexers via HTTP Event Collector (HEC).
  • Flow collector.

Splunk_TA_stream (UF) test environment

Splunk_TA_stream (UF) tests were run with workloads up to 1 Gbps maximum. HEC is recommended for higher bandwidth traffic.

Test hardware

CentOS 6.7 (64-bit).
Dual Intel Xeon E5-2650 CPUs (16 2.0Ghz cores; 32 cores total).
164 GB RAM.

streamfwd.conf configuration

[streamfwd]
ipAddr = 0.0.0.0
logConfig = streamfwdlog.conf
port = 8889
processingThreads = 4
streamfwdcapture.0.interface = eth0
dedicatedCaptureMode = 0

Stream configuration

The universal forwarder runs with the default stream capture configuration.

Independent Stream forwarder (HEC) test environment

All independent Stream forwarder test environments use the same hardware configuration.

The only difference in the test setup is the list of streams enabled.

Test hardware

Independent streamfwd tests are run on the following server:

CentOS 6.7 (64-bit).
Dual Intel Xeon E5-2698 v3 CPUs (16 2.3Ghz cores; 32 cores total).
64 GB RAM.

streamfwd.conf configuration

[streamfwd]
ipAddr = 0.0.0.0
processingThreads = 4
dedicatedCaptureMode = 1
streamfwdcapture.0.interface = 0000:05:00.0
streamfwdcapture.1.interface = 0000:05:00.1

Stream configurations

Independent Stream forwarder streamfwd (HEC) tests measure performance on four different stream configurations. These configurations determine how much traffic is sent from streamfwd to the indexers, and how deeply the packets are inspected by streamfwd to extract events.

Configuration Events forwarded to indexers Packet inspection level
Default configuration Aggregate Deep
HTTP Raw Events Raw Events Deep
TCP/UDP Raw Events Raw Events Shallow
TCP/UDP Aggregation Aggregate Shallow

Default configuration

The default configuration is what comes out of the box with a fresh install of Stream. All streams that start with Splunk_* are enabled and all other streams that forward raw events are disabled. The Splunk_* streams create an aggregate of events in various streams so that users can estimate how much indexer capacity will be taken by Stream when they turn forwarding of various raw events on.

HTTP raw events

In this configuration, only HTTP raw events are enabled. However, since HTTP is a level 7 protocol, it must maintain state across packets to create HTTP events of interest.

TCP/UDP raw events

In this configuration, only TCP and UDP raw events are enabled. This looks no higher than level 4 of the network stack and so does not need to do deeper analysis, but sends information regarding all the raw packets that it gets.

TCP/UDP aggregation

In this configuration, we calculate the number of bytes transferred for each source IP address (src_ip) for TCP and UDP protocols. The aggregation is calculated every 30 seconds. This looks no higher than level 4 of the network stack so deeper analysis is not required.

Flow collector test environment

Test hardware

NetFlow collector tests are run on the following server:

 CentOS 6.7 (64-bit).
 Dual Intel Xeon E5-2698 v3 CPUs (16 2.3Ghz cores; 32 cores total).
 64 GB RAM

streamfwd.conf configuration

[streamfwd]
ipAddr = 0.0.0.0
processingThreads = 4
dedicatedCaptureMode = 0
httpRequestSenderThreads = 4
httpRequestSenderConnections = 40
netflowReceiver.0.port = 9996
netflowReceiver.0.protocol = udp
netflowReceiver.0.decoder = netflow
netflowReceiver.0.ip = 172.18.1.4
netflowReceiver.0.decodingThreads = 32

For Flow collector test results and methodology, see Flow collector test results in this manual.

PREVIOUS
Use Stream configuration templates
  NEXT
Splunk_TA_stream (UF) test results - default configuration

This documentation applies to the following versions of Splunk Stream: 7.0.0, 7.0.1


Was this documentation topic helpful?

Enter your email address, and someone from the documentation team will respond to you:

Please provide your comments here. Ask a question or make a suggestion.

You must be logged into splunk.com in order to post comments. Log in now.

Please try to keep this discussion focused on the content covered in this documentation topic. If you have a more general question about Splunk functionality or are experiencing a difficulty with Splunk, consider posting a question to Splunkbase Answers.

0 out of 1000 Characters