Can I add my own protocols?
No. Splunk Stream does does not provide a mechanism for adding protocols.
How do I direct traffic from Splunk_TA_stream to a specific index?
You can modify
Splunk_TA_stream/local/ to specify an index.
Note: This applies to all traffic that the particular instance of
Can I direct data to specific indices based on protocol?
No. Currently Splunk Stream does not let you direct data to different indices based on protocol. You can however set up this functionality using
transforms.conf files. For instructions, see Route specific events to a different index.
Can I configure endpoints to listen for specific protocols?
Yes. You can configure stream filters to listen for specific protocols on an endpoint. For example, you can use s_ip (source_ip), which is a common flow attribute, to filter for DNS traffic only on a DNS server. (Filtering by hostname is not supported.)
Note: There is a chance of duplication if the endpoints can see each other's traffic (in other words, the network switch is not restricting traffic to just those packets destined for the endpoint).
A more advanced configuration involves deploying renamed copies of
Splunk_TA_stream and using the Deployment Server to control which endpoints receive which copy. In this case, the renamed
Splunk_TA_stream must have their
etc/apps/local/inputs.conf modified to point to the correct parent app.
Caution: This is a highly custom configuration. We strongly recommended that you consult Splunk Professional Services before you implement this type of configuration.
Why is Splunk_TA_stream installed on the search head by default?
Splunk_TA_stream is installed on search heads by default in support of single instance deployments.
Splunk_TA_stream is also installed in
$SPLUNK_HOME/etc/deployment-apps by default. This facilitates use of the deployment server, which can automatically deploy
Splunk_TA_stream to any universal forwarders that you might add to a distributed deployment.
Can I stop Splunk_TA_stream on my search head from capturing data?
Yes. You can use the sc_ip field to filter out stream data on the search head. Or you can remove
Splunk_TA_stream from the search head.
Can Stream capture uni-directional traffic (ingress or egress only)?
No. Stream must see the full TCP connection handshake (and shutdown) to properly determine which is the request and which is the response.
Where on the TA do I set the URL to pull the configuration from splunk_app_stream?
Splunk_TA_stream communicates at regular intervals with
splunk_app_stream at a specified URL. If the TA detects a configuration change, it sends a GET request to
splunk_app_stream to retrieve the updated configuration. The URL of
splunk_app_stream is specified in
Splunk_TA_stream/local/inputs.conf. See How
streamfwd communicates with
Can Stream read PCAP files?
Yes. Stream lets you read PCAP files and send structured PCAP data to indexers using the
./streamfwd -r foo.pcap -s <host><server>.
Can Stream send raw PCAP file data into Splunk Enterprise?
No. The PCAP data that
streamfwd sends to Splunk indexers is structured event data, not raw packet data. See Send PCAP data
Can Stream decrypt packets and application data?
Yes. You can use an SSL private key to decrypt data that the
streamfwd binary captures, provided that the data is encrypted using an RSA cipher that uses the same private key.
Can Stream decrypt Diffie-Hellman (SSL key) traffic?
No. There is no way to capture Diffie-Hellman traffic, regardless if the
streamfwd binary is collecting data from a TAP or running on the host itself.
Can I use Chef, Puppet, and other utilities to deploy and manage Stream configuration files?
Yes. You can use Chef, Puppet, and other utilities to push the
streamfwd binary out to universal forwarders.
streamfwd binary must maintain a connection with
splunk_app_stream to retrieve the stream configuration. So in a Deployment Server + Stream Forwarder scenario we must actively maintain a connection from the universal forwarder (via Deployment Client mechanism, port 8089 by default on the Splunk host) and the
Splunk_TA_stream (port 8000 by default on the
splunk_app_stream instance). In a Puppet, etc. scenario, we must still maintain an active connection from the endpoint to the App for Stream host.
Why won't the streamfwd process start up?
I see the following complaint in the in the forwarder's splunkd.log file:
10-07-2014 16:11:26.140 -0400 INFO ModularInputs - Introspection setup completed for scheme "streamfwd".
10-07-2014 16:11:27.029 -0400 INFO ModularInputs - No stanzas found for scheme "streamfwd" in inputs.conf at script (re)start.
10-07-2014 16:11:27.034 -0400 INFO ExecProcessor - New scheduled exec process: /opt/splunkforwarder/etc/apps/Splunk_TA_stream/linux_x86_64/bin/streamfwd
10-07-2014 16:11:32.601 -0400 ERROR ExecProcessor - message from "/opt/splunkforwarder/etc/apps/Splunk_TA_stream/linux_x86_64/bin/streamfwd" log4cplus:ERROR Unable to open file: /opt/splunk/var/log/splunk/streamfwd.log
There is currently an assumption made at install time that the copy of
Splunk_TA_stream installed in
deployment-apps will land on a system that has the same directory structure as the source system. The correct approach is to modify
deployment-apps/Splunk_TA_stream/default/streamfwdlog.conf to reflect the correct path of the destination forwarders and then redeploy the app.
Everything is set up correctly, but I don't see any events. What's wrong?
streamfwd binary communicates with
splunk_app_stream at regular intervals to retrieve its configuration. You can find the URL of
splunk_app_stream that is used for this communication at
$SPLUNK_HOME/etc/apps/Splunk_TA_stream/local/inputs.conf. If you are not receiving stream events, make sure that no firewall rules are blocking access to the
2. If the Stream Forwarders fail to send data after upgrade, you may see messages similar to this one:
WARN  (HTTPRequestSender.cpp:1485) stream.SplunkSenderHTTPEventCollector - (#7) TCP connection failed: Connection refused
To resolve this, first verify that the Stream forwarder is correctly configured. Then go to the Stream Forward App and update your HEC configuration:
- In the Stream App, open the Distributed Forwarder Management page.
- Select "Install Stream Forwarders".
- Verify the curl command is the same one running on the Stream Forward App.
- Turn off the HEC Autoconfig option.
- Update the Endpoint URLs by manually typing in the HEC (HF or Indexer) URL.
Splunk Stream REST API reference
This documentation applies to the following versions of Splunk Stream™: 7.1.2, 7.1.3, 7.2.0