Configure Stream forwarder
There are two types of configuration that apply to Stream forwarder:
- Configure Streams: The Configure Streams UI lets you create and configure the specific network data "streams" that you want to capture. These stream configurations are sent at regular intervals to the
streamfwd
binary inSplunk_TA_stream
where data capture actually occurs. You can access the Configure Streams UI in the Splunk App for Stream main menu in Splunk Web. For more information, see "Configure Streams" in the Splunk App for Stream User Manual.
- streamfwd.xml: The
streamfwd.xml
configuration file lets you specify system-level data capture parameters for thestreamfwd
binary. You can accessstreamfwd.xml
from the command line at$SPLUNK_HOME/etc/apps/Splunk_TA_stream/default
. For more information, see Configure streamfwd.xml.
Verify streamfwd can communicate with splunk_app_stream
Before you configure stream data capture in the Configure Streams UI, make sure that the streamfwd
binary can communicate with splunk_app_stream
. To do so, confirm that the local inputs.conf
file for Splunk_TA_stream
contains the correct location of your splunk_app_stream
installation.
1. Open $SPLUNK_HOME/etc/apps/Splunk_TA_stream/local/inputs.conf
.
2. Confirm that the [streamfwd://streamfwd]
stanza contains the correct location (URI) of your splunk_app_stream
installation.
For example:
[streamfwd://streamfwd] splunk_stream_app_location = https://localhost:8000/en-us/custom/splunk_app_stream/ disabled = 0
Specify the location of splunk_app_stream
You specify the location (URI) of your splunk_app_stream
installation when you create a Wire Data modular input for your Stream data in Splunk Enterprise. The streamfwd
binary uses this location to communicate with the splunk_app_stream over HTTP port 8000. splunk_app_stream
stores the location that you specify in the local inputs.conf
file of Splunk_TA_stream
. See How streamfwd communicates with splunk_app_stream.
1. Go to Settings > Data Inputs > Wire Data.
2. Click New.
3. Enter a Name for your Wire Data modular input. For example "streamfwd."
4. Enter the URI (including the full path) to your splunk_app_stream
installation. The URI must specify protocol, server, and port variables in the following syntax:
<protocol>://<server>:<port>/en-us/custom/splunk_app_stream/
For example:
http://localhost:8000/en-us/custom/splunk_app_stream
Note: The splunk_app_stream
location URI only supports http
and https
protocols.
Note: If you enable SSL for the Stream forwarder, you must change the URI path to specify https
. If you change the http port, you must change the URI path to specify the new port.
Configure Stream forwarder identifier
If using a deployment server, be aware that when you set or modify the stream_forwarder_id
of a specific Stream forwarder while a streamfwd process is running, you must restart the universal forwarder for the changes to the stream_forwarder_id
to apply. Further, note that multiple Stream forwarder deployments can share the same stream_forwarder_id
.
Configure streamfwd.xml
The streamfwd.xml
configuration file lets you specify system-level parameters for the streamfwd
binary. You can configure streamfwd.xml
to listen on specific IP addresses and ports, enable SSL, redirect log files, collect network events, and specify network interfaces.
streamfwd.xml
is included with Splunk_TA_stream
and is located in:
$SPLUNK_HOME/etc/apps/Splunk_TA_stream/default
.
Caution: Do not edit the streamfwd.xml
file in the $SPLUNK_HOME/etc/apps/Splunk_TA_stream/default
directory. To edit streamfwd.xml
, copy the streamfwd.xml
to $SPLUNK_HOME/etc/apps/Splunk_TA_stream/local
and make your edits there .
Basic configuration
streamfwd.xml
is configured by default to listen for traffic on all available network interfaces.
<?xml version="1.0" encoding="UTF-8"?> <CmConfig xmlns="http://purl.org/cloudmeter/config" version="6.0.0"> <Port>8889</Port> <UIDirectory>../../ui</UIDirectory> <DataDirectory>../../data</DataDirectory> <LogConfig>streamfwdlog.conf</LogConfig> </CmConfig>
streamfwd.xml
accepts these basic configuration parameters:
Tag | Description |
---|---|
<DataDirectory> | Location of failover and other data files |
<DefaultVocabularyPath> | Location of default vocabulary files (do not change) |
<Group> | Name of group the streamfwd process runs as
|
<IPAddr> | IP address that the Stream Forwarder listens on |
<LocalVocabularyPath> | Location of custom vocabulary files (do not change) |
<LogConfig> | Configuration file to use for logging |
<Port> | TCP port that the Stream Forwarder listens on (use "0" to disable) |
<UIDirectory> | Location of user interface files (do not change) |
<User> | Name of user the streamfwd process runs as
|
Advanced configuration
The streamfwd.xml
file accepts these advanced configuration options.
Caution: Do not modify these options unless your Splunk Support representative advises you to do so.
Tag | Description | Value type | Default value |
---|---|---|---|
<ClientIpSslHashBytes> | Defines number of client IP octets to use for SSL processor thread hash algorithm. (min value = 0; max value = 4) Applies only if you have _disabled_ <UseGlobalSSLSessionKeyCache> | client IP octets | 2 |
<DuplicatePacketWindow> | Defines number of packets cached in memory (using a rolling window) to detect duplicate packets. Set this to a value greater than zero to enable automatic deduplication of network packets. | packets cached in-memory | 0 |
<HideCreditCardNumbers> | Masks credit card numbers. Set to false to show all credit card numbers. | boolean | true |
<MapSslServers> | Set to false to disable automatic caching of encrypted versus unencrypted services. | boolean | true |
<MaxEventQueueSize> | Defines maximum number of events queued for delivery to Splunk. | events | 10000 |
<MaxFieldSize> | Defines maximum size of content field. | bytes | 10240 |
<MaxPacketQueueSize> | Defines maximum size for each processing threads' packet queue. | packets | 250000 |
<MaxTcpReassemblyPacketCount> | Defines maximum number of TCP packets in reassembly queue per processing thread. | TCP packets | 500000 |
<MaxTcpSessionCount> | Defines maximum number of concurrent TCP/UDP flows per processing thread. | TCP/UDP flows | 50000 |
<PcapBufferSize> | Defines buffer size for each network device. Increase the number of bytes if you see dropped packets. | bytes | 33554432 |
<PingInterval> | Modifies the ping server interval. | seconds | 5 |
<ProcessingThreads> | Defines number of threads to use for processing network traffic. | threads | 1 |
<QueueEventDelivery> | Determines thread use for processing captured events. Set to true to use a separate thread for processing. | boolean | false |
<SessionKeyTimeout> | Indicates idle time before SSL session keys expire. | seconds | 3600 |
<TcpConnectionTimeout> | Indicates idle time before TCP/UDP flows expire. | seconds | 180 |
<UseGlobalSSLSessionKeyCache> | Enables sharing of SSL cache across processing threads. Set to true to share. | boolean | false |
<UsePacketMemoryPool> | When set to true, Stream Forwarder uses a pool allocator to allot memory for storing network packets. Because the pool allocator does not release unused memory back to the operating system, setting this parameter to true may result in high memory usage. Set to true only when Splunk App for Stream is running on a dedicated capture server that processes large traffic volumes. | boolean | false |
Disable stream forwarder admin interface
By default, the Stream Forwarder admin interface is enabled, listening on TCP port 8889. To disable the interface:
1. Go to $SPLUNK_HOME/etc/apps/Splunk_TA_stream/local
.
2. Open the streamfwd.xml
configuration file and change <Port>8889</Port>
to <Port>0</Port>
.
Use TcpServer parameter to specify TCP servers
Stream forwarder automatically detects the client and server endpoints when it captures the beginnings of TCP connections. If it starts capturing traffic after establishing a TCP connection, Stream forwarder normally assumes that the sender of the first packet it sees is the client.
You can modify this behavior by inserting <TcpServer>
clauses that define the endpoints of specific TCP servers. If the sender of a packet matches the endpoint, Stream forwarder correctly categorizes it as a server response packet.
TCP Server element examples
Example 1: Single HTTP server endpoint
<TcpServer> <Address>192.168.1.102</Address> <Port>80</Port> </TcpServer>
Example 2: Wildcard endpoint
<TcpServer> <Address>192.168.1.0</Address> <AddressWildCard>255.255.255.0</AddressWildCard> <Port>80</Port> </TcpServer>
Use SSLServer element to specify SSL servers
Stream forwarder detects endpoint encryption, and attempts to decrypt SSL sessions using the available private keys. Optionally, you can explicitly define the traffic as encrypted or decrypted by inserting <SSLServer>
clauses.
<SSLServer> <Address>192.168.1.102</Address> <Port>443</Port> </SSLServer>
Use Capture element to specify network interfaces
By default streamfwd.xml
listens for traffic on all available network interfaces. To restrict data capture to specific network interfaces, you must insert a separate XML <Capture></Capture>
clause into the streamfwd.xml
file for each network interface on which you want to capture data.
You can specify multiple Capture elements in a single streamfwd.xml
file, and each Capture element can have its own <Filter> that applies to the specified network interface only.
For example, to specify a network interface on *nix:
<Capture> <Interface>eth0</Interface> <Offline>false</Offline> <Filter>tcp port 80</Filter> </Capture>
The <Capture>
element supports the following options:
Tag | Description |
---|---|
<Interface> | Should be set to the path of your pcap file |
<Offline> | True means use pcap, false means <Interface> is a network device name |
<Filter> | Lets you set a BPF (Berkeley Packet Filter) for kernel-level packet filtering. The value of this tag must comply with BPF syntax. Only one <Filter> tag per <Capture> element is supported. |
<Repeat> | True means to play back the pcap file repeatedly for continuous load |
<SysTime> | True means to use the system time for packet timestamps |
<BitsPerSecond> | Rate limiter, defaults to 10 Mbps if undefined and <Repeat> is true |
Specify a network interface on Windows
The following Capture clause specifies a Windows network interface:
<Capture> <Interface>\Device\NPF_{D6995D00-B75C-48DB-99AA-69F0150126BC}</Interface> <Offline>false</Offline> <Filter>tcp port 80</Filter> </Capture>
On Windows, you can substitute the <Interface>
or <InterfaceRegex>
name (such as <Interface>\Device\NPF_{D6995D00-B75C-48DB-99AA-69F0150126BC}</Interface>
) with the <Alias>
or <Description>
value returned by the --iflist
command line option.
For example, <Interface>Local Area Connection 2</Interface>
or <InterfaceRegex>Local Area.*</IntefaceRegex>
.
For more information, see "List network interfaces on Windows and Linux" in this manual.
Capture element examples
Example 1: Configure streamfwd.xml to include local loopback capture
Stream Forwarder by default does not capture traffic that originates and terminates on the same machine. You can enable capture of this "local loopback" traffic using a Capture element in the configuration file:
<Capture> <InterfaceRegex>(en|eth|lo)[0-9]*</InterfaceRegex> </Capture>
The <InterfaceRegex>
element instructs streamfwd.xml
to expand and enumerate the interfaces that are actually available on the host machine, and dynamically generates internal configurations for each network interface that matches the regular expression.
Example 2: Configure streamfwd.xml for use across multiple systems
You might want to maintain a master copy of streamfwd.xml
that you can reuse across multiple systems that have different network device names. The following streamfwd.xml
configuration listens on all matching interfaces found.
<Capture> <InterfaceRegex>.*</InterfaceRegex> </Capture>
Note that this configuration may generate startup warnings for any devices that do not support passive data capture.
Example 3: Capture data on specific network interfaces
In this example, on a system with 8 network interfaces, streamfwd.xml
would listen only for tcp port 80 traffic on only two of those interfaces (4 and 5):
<Capture> <InterfaceRegex>eth[45]</InterfaceRegex> <Offline>false</Offline> <Filter>tcp port 80</Filter> </Capture>
Example 4: Use pcap file instead of network interface
You can also use a previously generated pcap file instead of an actual network interface, using this variation of the <Capture> element.
<Capture> <Interface>/tmp/data.cap</Interface> <Offline>true</Offline> <Filter>tcp port 80</Filter> <Repeat>true</Repeat> <SysTime>true</SysTime> <BitsPerSecond>10000000</BitsPerSecond> </Capture>
Example 5: Add <Capture> element to streamfwd.xml configuration file
When you add your <Capture> element to your streamfwd.xml
configuration, place it inside the <CmConfig> element.
<?xml version="1.0" encoding="UTF-8"?> <CmConfig xmlns="http://purl.org/cloudmeter/config" version="6.4.0"> <Port>8889</Port> <UIDirectory>../ui</UIDirectory> <DataDirectory>../data</DataDirectory> <LogConfig>streamfwdlog.conf</LogConfig> <Capture> <InterfaceRegex>eth[45]</InterfaceRegex> <Offline>false</Offline> <Filter>tcp port 80</Filter> </Capture> <ProcessingThreads>4</ProcessingThreads> <SessionKeyTimeout>30</SessionKeyTimeout> <TcpConnectionTimeout>30</TcpConnectionTimeout> <UseGlobalSSLSessionKeyCache>false</UseGlobalSSLSessionKeyCache> <UsePacketMemoryPool>false</UsePacketMemoryPool> </CmConfig>
Configure universal forwarder for use with Stream | Add SSL keys for decryption |
This documentation applies to the following versions of Splunk Stream™: 6.4.0, 6.4.1, 6.4.2
Feedback submitted, thanks!