Configure Stream forwarder
There are two types of configuration that apply to Stream forwarder:
- inputs.conf: The
inputs.conf
file insideSplunk_TA_stream/local
must contain the location (URI) of yoursplunk_app_stream
installation. Thestreamfwd
binary uses this location to retrieve the specific stream capture configurations (protocols, fields, aggregation types, and so on) that you define in the Configure Streams UI. For more information, see Configure Streams in the Splunk Stream User Manual. - streamfwd.conf:
Splunk_TA_stream/local/streamfwd.conf
lets you specify system-level data capture parameters for thestreamfwd
binary. For more information, see Configure streamfwd.conf on this page.
Verify location of splunk_app_stream in inputs.conf
Before you set up stream data capture using the Configure Streams UI, make sure that Splunk_TA_stream/local/inputs.conf
is configured to communicate with splunk_app_stream
.
- Open
$SPLUNK_HOME/etc/apps/Splunk_TA_stream/local/inputs.conf
. - Confirm that the
[streamfwd://streamfwd]
stanza contains the correct location (URI) of yoursplunk_app_stream
installation. For example:[streamfwd://streamfwd] splunk_stream_app_location = https://localhost:8000/en-us/custom/splunk_app_stream/ disabled = 0
For more information, see How Splunk_TA_stream communicates with splunk_app_stream in this manual.
Note: The splunk_app_stream
URI supports http
and https
protocols. If you enable SSL for the streamfwd
binary, you must change the URI path to specify https
. If you change the http port, you must change the URI path to specify the new port.
Configure Stream forwarder identifier
If you are using a deployment server, when you set or modify the stream_forwarder_id
of a specific Stream forwarder while a streamfwd process is running, you must restart the universal forwarder for the changes to apply to the stream_forwarder_id
.
Note: Multiple Stream forwarder deployments can share the same stream_forwarder_id
.
Configure streamfwd.conf
The streamfwd.conf
configuration file lets you specify system-level parameters for the streamfwd
binary. You can configure streamfwd.conf
to listen on specific IP addresses and ports, enable SSL, redirect log files, collect network events, and specify network interfaces.
The streamfwd.conf
configuration file comes with both Splunk_TA_stream
and indpendent Stream forwarder, and is located in $SPLUNK_HOME/etc/apps/Splunk_TA_stream/default/
and /opt/streamfwd/default/
, respectively.
Caution: Do not change or copy the streamfwd.conf
file located in the default
directory. To modify the streamfwd.conf
file, create a new version of streamfwd.conf
in the local
directory. For more information, see Configuration file directories in the Splunk Enterprise Admin Manual.
Enable loopback capture
Stream forwarder does not capture local loopback traffic by default. To capture loopback traffic, add the appropriate streamfwdcapture
parameter to streamfwd.conf
, as shown in streamfwdcapture Example 1 on this page.
Streamfwd.conf parameters
The streamfwd.conf
configuration file accepts these parameter.
Parameter | Description | Value type | Default value |
---|---|---|---|
clientIpSslHashBytes | Defines number of client IP octets to use for SSL processor thread hash algorithm. (min value = 0; max value = 4) Applies only if you have _disabled_ useGlobalSSLSessionKeyCache | client IP octets | 2 |
dedicatedCaptureMode | Enables dedicated capture mode, which provides support for 10Gbps capture on compatible network interfaces. To enable dedicated capture mode, add dedicatedCaptureMode = 1 to streamfwd.conf. | boolean | 0 (false) |
duplicatePacketWindow | Defines number of packets cached in memory (using a rolling window) to detect duplicate packets. Set this to a value greater than zero to enable automatic deduplication of network packets. | packets cached in-memory | 0 |
hideCreditCardNumbers | Masks credit card numbers. Set to false to show all credit card numbers. | boolean | true |
mapSslServers | Set to false to disable automatic caching of SSL server certificates to corresponding servers's IP addresses. | boolean | true |
maxEventQueueSize | Defines maximum number of events queued for delivery to Splunk. | events | 10000 |
maxFieldSize | Defines maximum size of content field. | bytes | 10240 |
maxPacketQueueSize | Defines maximum size for each processing threads' packet queue. Must be a power of 2 for dedicated capture mode. | packets | 262144 |
maxTcpReassemblyPacketCount | Defines maximum number of TCP packets in reassembly queue per processing thread. | TCP packets | 500000 |
maxTcpSessionCount | Defines maximum number of concurrent TCP/UDP flows per processing thread. | TCP/UDP flows | 50000 |
pcapBufferSize | Defines buffer size for each network device. Increase the number of bytes if you see dropped packets. | bytes | 33554432 |
pingInterval | Modifies the ping server interval. | seconds | 5 |
processingThreads | Defines number of threads to use for processing network traffic. | threads | 2 |
sessionKeyTimeout | Indicates idle time before SSL session keys expire. | seconds | 3600 |
sslServer | Allows directly specify IP addresses/ports targeted for SSL decryption. | ||
streamfwdcapture | Restricts data capture to specified network interfaces | ||
tcpConnectionTimeout | Indicates idle time before TCP/UDP flows expire. | seconds | 180 |
tcpServer | Defines endpoints for TCP servers | ||
useGlobalSSLSessionKeyCache | Enables sharing of SSL cache across processing threads. Set to true to share. | boolean | false |
usePacketMemoryPool | When set to true, Stream forwarder uses a pool allocator to allot memory for storing network packets. Because the pool allocator does not release unused memory back to the operating system, setting this parameter to true may result in high memory usage. Set to true only when Stream forwarder is running on a dedicated capture server that processes large traffic volumes. | boolean | false |
Note: For a complete list of streamfwd.conf
parameters, see streamfwd.conf.spec
in $SPLUNK_HOME/etc/apps/Splunk_TA_stream/README
.
Use tcpServer to specify endpoints
Stream forwarder automatically detects the client and server endpoints when it captures the beginnings of TCP connections. If it starts capturing traffic after establishing a TCP connection, Stream forwarder normally assumes that the sender of the first packet it sees is the client.
You can modify this behavior by using the tcpServer
parameter to define the endpoints of specific TCP servers. If the sender of a packet matches the endpoint, Stream forwarder correctly categorizes it as a server response packet.
tcpServer examples
Example 1: Single HTTP server endpoint
tcpServer.N.address = 192.168.1.102 tcpServer.N.port = 80
Example 2: Wildcard endpoint
tcpServer.<N>.address = 192.168.1.0 tcpServer.<N>.addressWildCard = 255.255.255.0 tcpServer.<N>.port = 80
Use sslServer to specify encrypted/decrypted traffic
Stream forwarder detects endpoint encryption, and attempts to decrypt SSL sessions using the available private keys. Optionally, you can explicitly define the traffic as encrypted by adding sslServer
parameters.
sslServer.<N>.address = 192.168.1.102 sslServer.<N>.port = 443
Use streamfwdcapture to specify network interfaces
By default streamfwd.conf
listens for traffic on all available network interfaces. Using the streamfwdcapture
parameter you can restrict data capture to specific interfaces only.
The streamfwdcapture
parameter supports the following options:
Parameter | Description |
---|---|
streamfwdcapture.<N>.interface | Specify a network interface name or a path to a pcap file |
streamfwdcapture.<N>.interfaceRegex | Specify a regular expression to match multiple network interfaces |
streamfwdcapture.<N>.offline | True means use pcap, false means <Interface> is a network device name. Default is false |
streamfwdcapture.<N>.filter | Lets you set a BPF (Berkeley Packet Filter) for kernel-level packet filtering. The value of this tag must comply with BPF syntax. Only one Filter variable per streamfwdcapture parameter is supported. |
streamfwdcapture.<N>.repeat | True means to play back the pcap file repeatedly for continuous load |
streamfwdcapture.<N>.sysTime | True means to use the system time for packet timestamps instead of actual time timestamps from pcap file |
streamfwdcapture.<N>.bitsPerSecond | Rate limiter, defaults to 10 Mbps if undefined and <Repeat> is true |
To restrict data capture to specific network interfaces, you must insert a [streamfwd]
stanza into streamfwd.conf
. You can use streamfwdcatpure
parameters to specify multiple network interfaces in a single streamfwd.conf
file. For example, to specify two network interfaces - eth0 and eth1 -configured with different BPF filters on *nix:
[streamfwd] streamfwdcapture.0.interface = eth0 streamfwdcapture.0.filter = tcp port 80 streamfwdcapture.1.interface = eth1 streamfwdcapture.1.filter = udp port 53
Specify a network interface on Windows
The following specifies a Windows network interface:
streamfwdcapture.0.interface = \Device\NPF_{D6995D00-B75C-48DB-99AA-69F0150126BC} streamfwdcapture.0.offline = false streamfwdcapture.0.filter = tcp port 80
On Windows, you can substitute the streamfwdcapture.<N>.interface
or streamfwdcapture.<N>.InterfaceRegex
name (such as \Device\NPF_{D6995D00-B75C-48DB-99AA-69F0150126BC}
) with the <Alias>
or <Description>
value returned by the --iflist
command line option.
For example, streamfwdcapture.<N>.interface = Local Area Connection 2
or streamfwdcapture.<N>.InterfaceRegex = Local Area.*
.
For more information, see "List network interfaces on Windows and Linux" in this manual.
streamfwdcapture examples
Example 1: Configure streamfwd.conf to include local loopback capture
Stream forwarder by default does not capture traffic that originates and terminates on the same machine. You can enable capture of this "local loopback" traffic using the streamfwdcapture
parameter in the configuration file:
streamfwdcapture.<N>.interface = lo0
Note: you cannot use streamfwdcapture.<N>.interfaceRegex>
parameter to specify local loopback interfaces.
Example 2: Configure streamfwd.conf for use across multiple systems
You might want to maintain a master copy of streamfwd.conf
that you can reuse across multiple systems that have different network device names. The following streamfwd.conf
configuration listens on all matching interfaces found (excluding local loopback interfaces).
streamfwdcapture.<N>.interfaceRegex = .*
Note that this configuration may generate startup warnings for any devices that do not support passive data capture.
Example 3: Capture data on specific network interfaces
In this example, on a system with 8 network interfaces, streamfwd.conf
would listen only for tcp port 80 traffic on only two of those interfaces (4 and 5):
streamfwdcapture.<N>.interfaceRegex = eth[45] streamfwdcapture.<N>.offline = false streamfwdcapture.<N>.filter = tcp port 80
Example 4: Use pcap file instead of network interface
You can also use a previously generated pcap file instead of an actual network interface, using this variation of the streamfwdcapture parameter.
streamfwdcapture.<N>.interface = /tmp/data.cap streamfwdcapture.<N>.offline = true streamfwdcapture.<N>.filter = tcp port 80 streamfwdcapture.<N>.repeat = true streamfwdcapture.<N>.sysTime = true streamfwdcapture.<N>.bitsPerSecond = 10000000
For more information on how to ingest pcap files using the streamfwdcapture
parameter, see Ingest pcaps using streamfwd.conf in this manual.
Example 5: Add streamfwdcapture parameters to streamfwd.conf configuration file
You can add one or more streamfwdcapture
parameters to a [streamfwd] stanza to define capture behavior for a particular network interface. For example:
[streamfwd] streamfwdcapture.0.interfaceRegex = eth[45] streamfwdcapture.0.offline = false streamfwdcapture.0.filter = tcp port 80 streamfwdcapture.1.interface = eth0 streamfwdcapture.1.offline = false streamfwdcapture.1.filter = udp port 53
Stream Easy Setup | Configure file extraction |
This documentation applies to the following versions of Splunk Stream™: 7.1.0, 7.1.1
Feedback submitted, thanks!