Splunk Stream

Installation and Configuration Manual

Download manual as PDF

This documentation does not apply to the most recent version of StreamApp. Click here for the latest version.
Download topic as PDF

Configure Stream forwarder

About Stream forwarder configuration

There are two types of configuration that apply to Stream forwarder:

  • Streams Config: The Streams Config UI lets you select and configure the specific wire data protocols that you want streamfwd to capture. You can access the Streams Config UI inside the Splunk App for Stream through Splunk Web. For more information, see Streams Config.
  • streamfwd.xml: The streamfwd.xml configuration file lets you specify system-level data capture parameters for the streamfwd binary. You can access streamfwd.xml from the command line at $SPLUNK_HOME/etc/apps/Splunk_TA_stream/default. For more information, see Configure streamfwd.xml.

Note: To edit streamfwd.xml, you must copy the file to Splunk_TA_stream/local, and make your changes to the copy of streamfwd.xml in Splunk_TA_stream/local.

Verify streamfwd can communicate with splunk_app_stream

Before you configure data capture parameters for the streamfwd binary using the Stream Config UI, make sure that the streamfwd binary is configured to communicate with splunk_app_stream. To do so, confirm that the local inputs.conf file for Splunk_TA_stream contains the correct location of your splunk_app_stream installation.

1. Open $SPLUNK_HOME/etc/apps/Splunk_TA_stream/local/inputs.conf.

2. Confirm that the [streamfwd://streamfwd] stanza contains the correct location (URI) of your splunk_app_stream installation.

For example:

[streamfwd://streamfwd]
splunk_stream_app_location = https://localhost:8000/en-us/custom/splunk_app_stream/
disabled = 0

Specify the location of splunk_app_stream

You specify the location (URI) of your splunk_app_stream installation when you create a Wire Data modular input for your Stream data in Splunk Enterprise. The streamfwd binary uses this location to communicate with the splunk_app_stream over HTTP port 8000. splunk_app_stream stores the location that you specify in the local inputs.conf file of Splunk_TA_stream. See How streamfwd communicates with splunk_app_stream.

1. Go to Settings > Data Inputs > Wire Data.

2. Click New.

3. Enter a Name for your Wire Data modular input. For example "streamfwd."

4. Enter the URI (including the full path) to your splunk_app_stream installation. The URI must specify protocol, server, and port variables in the following syntax:

<protocol>://<server>:<port>/en-us/custom/splunk_app_stream/

For example:

http://localhost:8000/en-us/custom/splunk_app_stream

Note: The splunk_app_stream location URI only supports http and https protocols.

Note: If you enable SSL for the Stream forwarder, you must change the URI path to specify https. If you change the http port, you must change the URI path to specify the new port.

Configure streamfwd.xml

The streamfwd.xml configuration file lets you specify system-level parameters for the streamfwd binary. You can configure streamfwd.xml to listen on specific IP addresses and ports, enable SSL, redirect log files, collect network events, and specify network interfaces. streamfwd.xml also specifies default data extractionsfor HTTP protocol events.

streamfwd.xml is included with Splunk_TA_stream and is located in:

$SPLUNK_HOME/etc/apps/Splunk_TA_stream/default.

Caution: Do not edit the streamfwd.xml file in the $SPLUNK_HOME/etc/apps/Splunk_TA_stream/default directory. To edit streamfwd.xml, copy the streamfwd.xml to $SPLUNK_HOME/etc/apps/Splunk_TA_stream/local and make your edits there .

Basic configuration

streamfwd.xml is configured by default to listen for traffic on all available network interfaces.

<?xml version="1.0" encoding="UTF-8"?>
<CmConfig xmlns="http://purl.org/cloudmeter/config" version="6.0.0">
  <Port>8889</Port>
  <UIDirectory>../../ui</UIDirectory>
  <DataDirectory>../../data</DataDirectory>
  <LogConfig>streamfwdlog.conf</LogConfig>
</CmConfig>

streamfwd.xmlaccepts these basic configuration parameters:

Tag Description
<IPAddr> IP address that the Stream Forwarder listens on
<Port> TCP port that the Stream Forwarder listens on (use "0" to disable)
<User> Name of user the streamfwd process runs as
<Group> Name of group the streamfwd process runs as
<SSLKey> To enable Stream Forwarder passive SSL decryption, specify a PEM-encoded RSA private key file
<LogConfig> Configuration file to use for logging
<DataDirectory> Location of failover and other data files
<UIDirectory> Location of user interface files (do not change)
<DefaultVocabularyPath> Location of default vocabulary files (do not change)
<LocalVocabularyPath> Location of custom vocabulary files (do not change)

Advanced configuration

The streamfwd.xmlfile accepts these advanced configuration options.

Caution: Do not modify these options unless advised by your Splunk Support representative.

Tag Description
<MaxFieldSize> Maximum size of content field, in bytes (default=10240).
<ProcessingThreads> Number of threads to use for processing network traffic (default=1)
<MaxPacketQueueSize> Maximum size for each processing threads' packet queue (default=250000)
<PcapBufferSize> Buffer size for each network device, in bytes (default=33554432). Try increasing this if you see dropped packets.
<MaxEventQueueSize> Maximum number of events queued for delivery to Splunk (default=10000).
<SessionKeyTimeout> Idle time in seconds before SSL session keys are expired
<TcpConnectionTimeout> Idle time in seconds before TCP/UDP flows are expired
<MaxTcpSessionCount> Maximum number of concurrent TCP/UDP flows per processing thread (default=50000)
<MaxTcpReassemblyPacketCount> Maximum number of TCP packets in reassembly queue per processing thread (default=500000)
<DuplicatePacketWindow> Set this to a value greater than zero to enable automatic de-duplication of network packets. The value indicates the number of packets cached in memory (using a rolling window) to detect duplicate packets.
<UseGlobalSSLSessionKeyCache> Set to "true" to share SSL cache across processing threads
<HideCreditCardNumbers> Set to "false" to disable the automatic masking of credit card numbers
<QueueEventDelivery> Set this to "true" to enable the use of a separate thread for the processing of captured events
<MapSslServers> Set this to "false" to disable automatic caching of encrypted versus unencrypted services
<ClientIpSslHashBytes> Number of Client IP octets to use for SSL processor thread hash algorithm (if global ssl session key cache is disabled)
<PingInterval> Use this option to modify the ping server interval (in seconds). The default ping server interval is 5 seconds.
<Protocol> Setting this to "http" will add support for more advanced content extraction, but only HTTP traffic
<MaxRequestContentLength> Max number of bytes extracted from HTTP request content (requires <Protocol>http</Protocol>)
<MaxResponseContentLength> Max number of bytes extracted from HTTP response content (requires <Protocol>http</Protocol>)
<RawRequestHeaders> Set to "true" to enable extraction of raw HTTP request headers (requires <Protocol>http</Protocol>)
<RawResponseHeaders> Set to "true" to enable extraction of raw HTTP response headers (requires <Protocol>http</Protocol>)
<AllowUtf8Conversion> Set to "true" to enable UTF8 conversion of HTTP request/response content (requires <Protocol>http</Protocol>)
<AllowSearchingContentForCharset> Set to "true" to enable searching of content for charset (requires <Protocol>http</Protocol>)

Disable Stream Forwarder admin interface

By default, the Stream Forwarder admin interface is enabled, listening on TCP port 8889. To disable the interface:

1. Go to $SPLUNK_HOME/etc/apps/Splunk_TA_stream/local.

2. Open the streamfwd.xml configuration file and change <Port>8889</Port> to <Port>0</Port>.


Use TcpServer element to specify TCP servers

Stream forwarder automatically detects the client and server endpoints when it captures the beginnings of TCP connections. If it starts capturing traffic after establishing a TCP connection, Stream forwarder normally assumes that the sender of the first packet it sees is the client.

You can modify this behavior by inserting <TcpServer> clauses that define the endpoints of specific TCP servers. If the sender of a packet matches the endpoint, Stream forwarder correctly categorizes it as a server response packet.

TCP Server element examples

Example 1: Single HTTP server endpoint

<TcpServer>
    <Address>192.168.1.102</Address>
    <Port>80</Port>
</TcpServer>

Example 2: Wildcard endpoint

<TcpServer>
    <Address>192.168.1.0</Address>
    <AddressWildCard>255.255.255.0</AddressWildCard>
    <Port>80</Port>
</TcpServer>

Use SSLServer element to specify SSL servers

Stream forwarder detects endpoint encryption, and attempts to decrypt SSL sessions using the available private keys. Optionally, you can explicitly define the traffic as encrypted or decrypted by inserting <SSLServer> clauses.

<SSLServer>
    <Address>192.168.1.102</Address>
    <Port>443</Port>
</SSLServer>

Use Capture element to specify network interfaces

By default streamfwd.xml listens for traffic on all available network interfaces. To restrict data capture to specific network interfaces, you must insert a separate XML <Capture></Capture> clause into the streamfwd.xml file for each network interface on which you want to capture data.

You can specify multiple Capture elements in a single streamfwd.xml file, and each Capture element can have its own <Filter> that applies to the specified network interface only.

For example, to specify a network interface on *nix:

<Capture>
    <Interface>eth0</Interface>
    <Offline>false</Offline>
    <Filter>tcp port 80</Filter>
</Capture>

The <Capture> element supports the following options:

Tag Description
<Interface> Should be set to the path of your pcap file
<Offline> True means use pcap, false means <Interface> is a network device name
<Filter> Lets you set a BPF (Berkeley Packet Filter) for kernel-level packet filtering. The value of this tag must comply with BPF syntax. Only one <Filter> tag per <Capture> element is supported.
<Repeat> True means to play back the pcap file repeatedly for continuous load
<SysTime> True means to use the system time for packet timestamps
<BitsPerSecond> Rate limiter, defaults to 10 Mbps if undefined and <Repeat> is true

Specify a network interface on Windows

The following Capture clause specifies a Windows network interface:

<Capture>
    <Interface>\Device\NPF_{D6995D00-B75C-48DB-99AA-69F0150126BC}</Interface>
    <Offline>false</Offline>
    <Filter>tcp port 80</Filter>
</Capture>

On Windows, you can substitute the <Interface> or <InterfaceRegex> name (such as <Interface>\Device\NPF_{D6995D00-B75C-48DB-99AA-69F0150126BC}</Interface>) with the <Alias> or <Description> value returned by the --iflist command line option.

For example, <Interface>Local Area Connection 2</Interface> or <InterfaceRegex>Local Area.*</IntefaceRegex>.

For more information, see "List network interfaces on Windows and Linux" in this manual.

Capture element examples

Example 1: Configure streamfwd.xml to include local loopback capture

Stream Forwarder by default does not capture traffic that originates and terminates on the same machine. You can enable capture of this "local loopback" traffic using a Capture element in the configuration file:

<Capture>
    <InterfaceRegex>(en|eth|lo)[0-9]*</InterfaceRegex>
</Capture>

The <InterfaceRegex> element instructs streamfwd.xml to expand and enumerate the interfaces that are actually available on the host machine, and dynamically generates internal configurations for each network interface that matches the regular expression.

Example 2: Configure streamfwd.xml for use across multiple systems

You might want to maintain a master copy of streamfwd.xml that you can reuse across multiple systems that have different network device names. The following streamfwd.xml configuration listens on all matching interfaces found.

<Capture>
    <InterfaceRegex>.*</InterfaceRegex>
</Capture>

Note that this configuration may generate startup warnings for any devices that do not support passive data capture.

Example 3: Capture data on specific network interfaces

In this example, on a system with 8 network interfaces, streamfwd.xml would listen only for tcp port 80 traffic on only two of those interfaces (4 and 5):

<Capture>
    <InterfaceRegex>eth[45]</InterfaceRegex>
    <Offline>false</Offline>
    <Filter>tcp port 80</Filter>
</Capture>

Example 4: Use pcap file instead of network interface

You can also use a previously generated pcap file instead of an actual network interface, using this variation of the <Capture> element.

<Capture>
    <Interface>/tmp/data.cap</Interface>
    <Offline>true</Offline>
    <Filter>tcp port 80</Filter>
    <Repeat>true</Repeat>
    <SysTime>true</SysTime>
    <BitsPerSecond>10000000</BitsPerSecond>
</Capture>
PREVIOUS
Configure universal forwarder for use with Stream
  NEXT
Add SSL keys for decryption

This documentation applies to the following versions of Splunk Stream: 6.3.0, 6.3.1, 6.3.2


Was this documentation topic helpful?

Enter your email address, and someone from the documentation team will respond to you:

Please provide your comments here. Ask a question or make a suggestion.

You must be logged into splunk.com in order to post comments. Log in now.

Please try to keep this discussion focused on the content covered in this documentation topic. If you have a more general question about Splunk functionality or are experiencing a difficulty with Splunk, consider posting a question to Splunkbase Answers.

0 out of 1000 Characters