Use Splunk Stream to ingest Netflow and IPFIX data

NetFlow and IP Flow Information Export (IPFIX) are protocols that collect flow data from network devices. Use Splunk Stream to ingest Netflow and IPFIX data. You can use Splunk Stream to ingest Netflow and IPFIX data. Splunk Stream supports flow data sent over the UDP protocol.

Splunk Stream also lets you automatically pull proprietary Netflow configuration data from configuration apps. See Use Splunk Stream to ingest Netflow and IPFIX data

If you configure traffic with VLAN-ID, you must configure for both traffic directions. Otherwise you may experience a mismatch of VLANs in the TCP streams, which can cause queue overflow errors and traffic loss.

Configure indexers

Enable the Http_input receiver on your Splunk platform deployment's indexers:

1. Navigate to your splunk_httpinput directory.
   • For a single instance deployment navigate to $SPLUNK_HOME/etc/apps/splunk_httpinput/local/
   • For a distributed deployment navigate to $SPLUNK_HOME/etc/master-app/splunk_httpinput/local/
2. Create an inputs.conf file, if one does not already exist.
3. Open inputs.conf and add stanzas to enable receiving. For example:

   [http] 
   disabled = 0 
   port = 8088 
   dedicatedIoThreads = 8 
   
   [http://streamfwd] 
   disabled = 0
   index=main
   token = 152B6F2B-8A21-4850-A444-CB0646FD88BE 
   indexes=_internal,main
   

4. Save your changes and exit.
5. Restart your Splunk platform deployment.

(Optional) Modify Splunk Stream Forwarder and push to clustered indexers

For Splunk platform deployments that use indexer clustering, modify the Splunk Stream Forwarder:

1. Navigate to Splunk_TA_stream/default on your Splunk platform deployment.
2. Remove the following files, if they are present:
   • inputs.conf
   • inputs.conf.spec.
3. Push the modified Splunk_TA_stream to all indexers in your Splunk platform deployment.

Configure the Independent Stream Forwarder

Configure the Independent Stream Forwarder on your Splunk platform deployment.

1. On your deployment's Independent Stream Forwarder, navigate to streamfwd.conf.
2. Open streamfwd.conf and enable forwarding.

   [streamfwd] 
   httpEventCollectorToken = 152B6F2B-8A21-4850-A444-CB0646FD88BE 
   #(Match this with the token in the indexers) 
   ipAddr = 0.0.0.0 
   processingThreads = 4
   

3. In yourstreamfwd.conf file, add your deployment's NetFlow configuration.

   [streamfwd]
   
   httpEventCollectorToken = <GUID>
   
   indexer.0.uri= <HEC VIP>
   netflowReceiver.0.port = 9996
   netflowReceiver.0.decoder = netflow
   netflowReceiver.0.ip = 172.18.1.4
   netflowReceiver.0.decodingThreads = 16
   

4. Save your changes.
5. Restart your Splunk platform deployment.
6. Navigate to your independent Stream Forwarder's etc/sysctl.conf directory.
7. Adjust your kernel settings to increase buffer sizes for high-volume packet capture.

   sysctl -w net.core.rmem_default=33554432
   sysctl -w net.core.rmem_max=33554432
   sysctl -w net.core.netdev_max_backlog=10000
   

8. Reload the settings:

   /sbin/sysctl -p
   

9. Restart the streamfwd service:

   Service streamfwd restart
   

Configure search heads

1. Log in to the search head where the Splunk App for Stream is installed.
2. Navigate to the Splunk App for Stream, then click Configuration > Distributed Forwarder Management.
3. Click Create New Group.
4. Enter a name. For example, INFRA_NETFLOW.
5. Enter a description.
6. Click Next.
7. Enter the regular expression to match the streamforwarder_id or hostname of the "streamfwd" collectors that should participate in this group rule. Click Next.
8. Do not select any options. Click Finish.
9. Navigate to the Splunk App for Stream, then click Configuration > Configure Streams.
10. Click New Stream > Metadata.
11. Enter Name as INFRA_NETFLOW.
12. Select NetFlow as the protocol.

The NetFlow option works for NetFlow, sFlow, jFlow, and IPFIX protocols.

13. Enter a description then click Next.
14. Select No in the Aggregation box then click Next.
15. (Optional) Deselect any fields that do not apply to your use case then click Next.
16. (Optional) Develop filters to reduce noise from high traffic devices then click Next.
17. Select the index for this collection and click Enable then click Next.
18. Select only the INFRA_NETFLOW group and Create_Stream.
19. Configure your NetFlow generator to send records to the new streamfwd.
20. Validate your results by searching the configured index on your Splunk platform deployment.

Configure Flow collector

Splunk Stream supports flow protocol data ingestion from network devices. If you have switches, routers, firewalls. or other elements that generate flow protocol data such as NetFlow and sFlow, you can configure your Splunk Stream Forwarder or Splunk Independent Stream Forwarder to support flow data ingestion.

Supported flow protocols

Stream supports collection of these flow protocols:

• NetFlow version 5, 9 and IPFIX.
• sFlow version 5
• jFlow

Splunk Stream supports flow data sent over the UDP protocol.

Best practices for scaling flow ingestion

When scaling flow protocol ingestion, consider these best practices:

• Use Independent Stream forwarder for best results. See Deploy Independent Stream forwarder.
• Configure Nginx or another load balancer to distribute load among indexer cluster nodes.
• Disable SSL on the HEC input, if appropriate. (Do not disable SSL if you are sending data to Splunk Cloud, or if other security considerations apply.)

Note: Both Splunk Stream Forwarder and Splunk Independent Stream Forwarder deployments support flow protocol collection. However, due to the limited ingestion capabilities of the Splunk for Stream Wire Data used by Splunk_TA_stream, we recommended using Splunk Stream Forwarder for low bandwidth or aggregated NetFlow capture only.

Configure flow data ingestion

To ingest flow data, configure streamfwd to receive data at a specific IP address and port and specify the flow protocol. To do this, add a set of flow configuration parameters to streamfwd.conf as follows:

1. Edit local/streamfwd.conf.
2. Add the following parameters to specify the the IP address to bind to, the port number to bind to, and the flow protocol.

   netflowReceiver.<N>.ip = <ip_address>
   netflowReceiver.<N>.port = <port_number>
   netflowReceiver.<N>.decoder = <flow_protocol>
   

   For example, to receive NetFlow and sFlow data at IP address 172.23.245.122 on port 9995 and 6343 respectively, configure streamfwd.conf as shown:

   [streamfwd]
   logConfig = streamfwdlog.conf
   port = 8889
   
   netflowReceiver.0.ip = 172.23.245.122
   netflowReceiver.0.port = 9995
   netflowReceiver.0.decoder = netflow
   
   netflowReceiver.1.ip = 172.23.245.122
   netflowReceiver.1.port = 6343
   netflowReceiver.1.decoder = sflow
   

   For high volume with NetFlow data, configure additional NetFlow processing threads as shown:

   netflowReceiver.0.decodingThreads = 4
   

3. Restart Splunk.

By default, the netflowReceiver.<N>.ip parameter binds to the first available IP address. There are no default values for netflowReceiver.<N>.port and netflowReceiver.<N>.decoder configuration parameters.

Configure proprietary element mapping

Splunk Stream supports mapping IPFIX proprietary elements to Stream forwarder vocabulary terms. For more details see https://docs.splunk.com/Documentation/StreamApp/8.1.0/DeployStreamApp/AutoinputNetflow.

Flow data search syntax

To run searches for NetFlow or sFlow protocol data use the following search syntax:

sourcetype=stream:netflow
sourcetype=stream:sflow

Create flow protocol streams

Once you configure streamfwd.conf for flow data ingestion, you can use the Configure Streams UI in the Splunk App for Stream to create NetFlow and sFlow protocol streams with unique field definitions. See Configure Streams.

How NetFlow event timestamps are calculated

If you have any of the following fields in a your NetFlow data, Stream Forwarder sets the Splunk timestamp field to be the value contained in the NetFlow flowStart* field and the Splunk endtime field value to be the value contained in the NetFlow flowEnd* field.

• flowStartSeconds
• flowEndSeconds
• flowStartMilliseconds
• flowEndMilliseconds
• flowStartMicroseconds
• flowEndMicroseconds
• flowStartNanoseconds
• flowEndNanoseconds

For NetFlow records that are not flow related, when observationTime* fields are available, Stream Forwarder populates the Splunk timestamp and endtime fields values from the NetFlow observationTime*.

If you have both flowStart* and observationTime* fields in your NetFlow data, Stream Forwarder sets the Splunk Search timestamp to be the NetFlow flowStart* value and the Splunk Search endtime field to contain the NetFlow observationTime* value.

If none of the above fields are present, and a NetFlow record has the following fields:

• "first switch"(flowStartSysUpTime)
• "last switch"(flowEndSysUpTime)
• "system uptime"
• "current device time in unix epoch"

then Stream Forwarder calculates the Splunk Search timestamp and endtime as follows:

• timestamp = ("device time in unix epoch" - "system uptime") + "first switched"(flowStartSysUpTime)
• endtime = ("device time in unix epoch" - "system uptime") + "last switched"(flowEndSysUpTime)