Automatically input data with Netflow proprietary configurations

Splunk Stream lets you automatically pull proprietary Netflow configuration data from configuration apps. To do this, you install a configuration app that Splunk Stream uses to automatically configure Netflow proprietary elements.

If you configure traffic with VLAN-ID, you must configure for both traffic directions. Otherwise you may experience a mismatch of VLANs in the TCP streams, which can cause queue overflow errors and traffic loss.

How Splunk Stream updates configurations

When you install a configuration app on your search head, Splunk Stream automatically collects the configuration data from your third party application using scripted inputs and then seamlessly provides decoding of Netflow/IPFIX proprietary data.

To accomplish this, Splunk Stream runs a scripted input that searches for data from your configuration apps. This input fetches the data from the configuration app, validates the configurations, and pushes the configurations into Splunk Stream. The input also checks for upgraded, modified, or deleted apps. Note that this process is only relevant for NetFlow data at this time.

This scripted input executes every 120 seconds and processes data from configuration apps that use the syntax splunk_app_stream_ipfix_appname. See Create configuration apps.

Once the configurations are successfully incorporated in Splunk Stream, the input behaves like any other built-in configuration.

Create configuration apps

A configuration app consists of the following files packaged as a tar file for upload with the syntax splunk_app_stream_ipfix_appname:

• Vocabulary.xml file (netflow.xml)
• Vocabulary mapping file (ipfixmap.conf)
• Stream file (netflow.json)

You create or obtain these files and place them in the default folder of the app for installation. The following resources can help you create or obtain a configuration app:

• Detailed instructions for creating an app can be found at https://dev.splunk.com/enterprise/docs/developapps/createapps/.
• Example configuration app files are provided at (internal link) in this topic.
• Consult your vendor-specific documentation for more about vendor-specific netflow proprietary elements.
• For help creating the configuration files you can reach out to Splunk support.
• If your Splunk Stream instance is in a cloud environment, your configuration app must be approved by Splunk. See https://docs.splunk.com/Documentation/SplunkCloud/8.2.2203/Admin/PrivateApps.

Configuration app file precedence and behavior

Modifying configuration apps can affect how Splunk Stream processes your configuration app, keep the following in mind when you modify a configuration app:

• When any configuration file is updated, the new changes are validated and pushed into the app. If the new changes cannot be validated, Stream forwarder keeps the previous unchanged version.
• If the vocabulary file is deleted, both dependent files (the Stream file and the ipfixmap.conf file) are deleted from Stream App configurations.
• If the Stream file or Conf file is deleted, the configuration in Splunk Stream is deleted.
• If the configuration app is deleted, all relevant configurations are deleted from Splunk Stream.
• Splunk layers files with the same name in both local and default folders. This means that if the same file is available in the default folder and the local folder, then both files get merged. If the same content is available in both the files then content which is present in the local file gets higher priority.

Creating configuration app files

These examples show one possible way to create the files required for configuration apps. For more about creating these files, see your third-party product documentation or reach out to support. If you are running Splunk Stream in a cloud environment, your app must be approved by dev ops: https://docs.splunk.com/Documentation/SplunkCloud/8.2.2203/Admin/PrivateApps.

Create a vocabulary file

The following is an example of how to create a vocabulary.xml using the netflow.xml file.

<Term id="netflow.httpRequestMethod"> <Type>string</Type> <Comment>HTTP request method associated with a flow</Comment> </Term>

Here's an example of the exact syntax you can user:

<?xml version="1.0" encoding="UTF-8"?>
<CmConfig xmlns="http://purl.org/cloudmeter/config" version="8.1.1">
   <Vocabulary id="cisco">
       <Term id="netflow.ingressoverlaysessionid">
           <Type>uint32</Type>
           <Comment>ingressOverlaySessionId</Comment>
       </Term>
       <Term id="netflow.egressoverlaysessionid">
           <Type>uint32</Type>
           <Comment>egressOverlaySessionId</Comment>
       </Term>
       <Term id="netflow.routingvrfservice">
           <Type>uint32</Type>
           <Comment>routingVrfService</Comment>
       </Term>
       <Term id="netflow.connectionid">
           <Type>uint64</Type>
           <Comment>connectionId</Comment>
       </Term>
   </Vocabulary>
</CmConfig>

Note the following:

• The file name should be ipfixmap.conf.
• Change the version and the Vocabulary id to match the suffix part of the name of the app. For example:

<?xml version="1.0" encoding="UTF-8"?>
<CmConfig xmlns="http://purl.org/cloudmeter/config" version="8.4.0">
   <Vocabulary id="mycompany">

• For termid, the value should directly align to a value that is found in the netflow.xml file.
• After netflowElement, for each new element, increment the number.
• If you are using IPFIX (netflow v10) then netflowElement.3.enterpriseid is required for each element.

Create a vocabulary mapping file

The purpose of the vocabulary mapping is to map the vocabulary terms in the xml file to the ipfix or netflow element id value.

Splunk Stream supports mapping of IPFIX proprietary elements to Stream forwarder vocabulary terms. IPFIX should include the enterpriseid of the element and the file must be named ipfixmap.conf.

This lets you add and specify proprietary flow elements as fields in NetFlow protocol stream configurations that you create in the Configure Streams UI. Ifyou need assistance creating this file, contact your Splunk support representative.

By creating a mapping file, you map Netflow proprietary elements to Stream vocabulary terms: For example:

netflowElement.0.id = 459
netflowElement.0.termid = netflow.httpRequestMethod

.

Create a stream definition

The following is an example of how you can add a new vocabulary term to the Netflow Stream:

   {
     "aggType": "value", 
     "desc": "HTTP request method",
     "enabled": true, 
     "name": "netflow_http_req_method",
     "term": "netflow.httpRequestMethod"
   }

Note the following:

• Include the file name this aligns to (netflow.json) and should follow the syntax in the example: splunk_app_stream_ipfix
• This file controls all the fields you will see in the Stream app when setting up a Stream. To extend the fields that are part of netflow collection, append to what is already in the /opt/splunk/etc/apps/splunk_app_stream/default/streams/netflow file. It is important that since this is JSON, commas for the list are in the correct place.
• Make sure the term is aligned to the termid used in the ipfixmap.conf and term id used in netflow.xml.
• The id towards the bottom needs to be a unique name and must be the same value of what is in the vocabulary id in the netflow.xml file. This needs to be unique across all Streams, meaning it cannot be netflow.