Streams Config

What is a stream?

A "stream" is a grouping of network event data that:

• has a network protocol;
• has protocol attributes;
• can be raw events or aggregate events;
• can be customized (cloned) to extract more granular event data.

About the Streams Config UI

The Streams Config UI lets you configure stream event capture for a wide variety of supported protocols. You can select protocol attributes and filter data capture with a high degree of specificity so that you capture only the event data that you require for analysis, while minimizing the quantity of data that Splunk must index.

The Streams Config UI lets you:

• Select the wire data protocols that you want to capture.
• Use filters to to define granular stream capture.
• Create custom (cloned) streams.
• Aggregate event data over user-defined intervals.
• View, enable, clone and delete ephemeral streams (in correlation with supported Splunk apps).

For details on how the streamfwd binary receives configuration data from the Streams Config UI, see How streamfwd communicates with splunk_app_stream.

Enable or disable event capture

To enable or disable event capture for specific protocols:

1. In the Splunk App for Stream menu, click Streams Config.

The Streams Config UI opens.

2. Select Stream Type: Permanent.

3. In the Group by menu, select how to group the Streams view: by Category, Application, or Protocol.

4. In the Streams window, click on the name of the protocol that you want to capture.

The configuration page opens for that specific protocol.

5. Select the status Enabled or Disabled button.

The selected protocol is now enabled/disabled for event capture.

Note: HTTP protocol is enabled by default.

Select protocol attribute fields

You can choose the specific fields that you want to capture for each stream protocol.

1. Select the Enable checkbox to add that field to the list of protocol events.

2. Click Save.

Add filters

You can add filters to a Stream based on conditions that compare a Term with a specific value. For example, for the term "http_method," you could add a condition that http_method must contain the value GET. If all (or any) http_method matches this condition, the stream event data passes through.

1. Click the Add button next to the term for which you want to create the filter condition.

2. Select a Comparison type from the drop-down menu.

3. Enter the Value that defines the condition of the comparison.

4. Click Save Changes.

Your new filter is added to the Stream configuration file and appears in the list of filters for that Stream.

Note: Filter comparison values can be strings or numeric values.

Create custom streams

You can create custom streams that provide additional granularity for network data capture. You can also add additional filters to custom streams and aggregate custom stream data over specific time intervals.

Clone streams

To create a custom stream, you must clone an existing stream. When you clone a stream, the app produces an exact duplicate of the original stream, including all enabled fields and existing filters.

1. In the main menu, click Streams.

This opens the main Streams Config page.

2. Click Clone.

The Clone Stream dialog appears.

3. Enter a Name and Description for your stream. Click OK.

A new Streams Config page opens for the cloned stream.

4. Click Enabled to enable capture for the cloned stream.

5. Click Save.

Your new custom stream appears in the list of streams on the main Streams Config page.

Aggregate events

Splunk App for Stream lets you aggregate events for custom (cloned) streams over specific time-intervals. The app groups events into aggregation buckets, with one bucket allocated for each unique collection of "Key" fields. At the end of the time-interval, the app emits an object that represents each bucket.

For example, if you choose to aggregate bytes_in over an interval of 60 seconds, the app aggregates the total number of bytes_in over a 60 second interval in a unique bucket.

Aggregate Types

Key: Fields that have aggregate type Key are used for grouping and are included in the stream objects.

Sum: The values of fields that have aggregate type Sum are added together for all events over the time-interval.

To aggregate events:

1. On the Streams Config page, click on the name of your custom stream.

The configuration page for that custom stream opens.

2. Select the Aggregated checkbox.

This changes the Stream Type to "agg_events" and disables all fields.

3. Enter the Time Interval (in seconds) for aggregated events.

4. Select the Enable checkbox for each field you want to aggregate.

5. Select Key to define the aggregate bucket. This gives you a separate bucket for each value of the term. For example, if you select Key for the field "protocol" the app creates a separate bucket for each Level 7 protocol name (http, ftp, etc.).

6. Select Sum to aggregate numeric values for a field.

If your aggregate event includes multiple Key fields, Splunk App for Stream looks for unique combinations of Key event values and creates a unique bucket for each combination.

For example, if you assign Key to the field "protocol" and you assign Sum to the field "bytes_in," the app creates a unique bucket for each protocol value (http, ftp, etc.), and sums the number of bytes_in over the defined interval for each bucket. If you also assign Key to the "src_ip" and "dest_ip" fields, the app creates a unique bucket for each permutation of protocol, source, and destination IP address.

Use Content Extraction

The Content Extraction feature lets you capture and index a subset of data from a protocol string field. You can use this feature to capture only the specific section of data that you require, without indexing all of the data in the field. Content extraction can also help you minimize the total volume of data that Splunk indexes, which can help conserve indexer capacity.

For example, some string fields, such as src_content dest_content, and cookies, contain long strings of data. In many cases, the entire string of data is not useful. You might however want to capture a smaller section of the data, such as a name, ID, account number and so on. Content extraction lets you capture the specific pieces of data that you require, without indexing extraneous data.

Create content extraction rule

The content extraction rule uses a regular expression to extract sections of data from the original field. The rule also lets you specify a capturing group match, which outputs either the first value that matches the regex, or the "list" of all values that match the regex.

The content extraction rule creates a new field that captures only that data specified by the rule. The original field is not modified.

1. In the Stream Config UI, click on the protocol that contains the original field from which you want to extract content.

2. In the Actions menu, select Extract New Field.

The Content Extraction dialog appears.

Note: It is not necessary for the original field to be enabled, and normally it should be disabled. If you would also like to index the original field, you should consider using field transformations instead.

3. Enter a name for the content extraction rule. For example, "readable_cookie."

4. Enter a description for the content extraction rule. For example, "Separate the cookies into name and value pair in a readable format."

5. In the Extraction Rule field, enter the regular expression for the content that you want to extract. For example, we can use following regular expression to extract a name and value pair from the cookie field:

(.*?)\=(.*?;)

Note: Stream uses Boost Perl Regular Expression syntax.

6. Under Match:

• Select First to return only the first value that matches the regex.
• Select All to return the "list" of all values that match the regex.

7. In the Extraction Format plain text box, enter the format for the extraction. For example, enter $1, $2 to return the first and second values that match the regex.

This image shows the complete content extraction rule for our "readable_cookie" example:

8. Click Save.

The new field appears in the Streams Config UI. The term "Extracted" appears in the Type column.

Splunk search results for the "readable_cookie" field in our example should appear similar to this:

Note: The <MaxEventQueueSize> option in streamfwd.xml determines the maximum number of events that Splunk_TA_stream can queue for delivery to Splunk indexers. By default <MaxEventQueueSize> supports 10k events. To increase or decrease the maximum event queue size, modify the value of <MaxEventQueueSize> in streamfwd.xml. See Configure streamfwd.xml

Use Stats Only mode

The Stats Only mode collects index volume stats on stream data, without sending the stream data to your indexers. These stats show you the amount of data that your Stream deployment is processing for each protocol, which can help you configure stream capture and determine your indexer requirements.

Note: Stream collects and indexes both stream data and stats for any enabled protocol by default.

To use Stats Only mode:

1. In the Action menu for the specific protocol, select Stats Only.

2. In the confirmation window, select Yes.

Use the Stream Stats dashboard to view index volume and other Stats Only mode data.

Note: Stats Only mode lets you collect stats from both enabled and disabled protocols.

Support for ephemeral streams

Ephemeral streams give users of external Splunk apps the ability to schedule stream capture for a user-defined period of time. This is useful if you want to capture and analyze a limited number of network events pertaining to specific network activity or transactions over a specific time interval.

While you must configure and schedule ephemeral stream capture from within the context of the external Splunk app, you can view and perform certain actions on existing ephemeral streams (such as enable/disable) inside the Streams Config UI.

How ephemeral streams work

Splunk apps that support ephemeral streams take advantage of the Stream REST API. Ephemeral streams are similar to normal streams, but ephemeral streams require two additional parameters, createDate and expirationDate, which specify stream creation and expiration, respectively, in epoch time. For example:

createDate: 1404259338
expirationDate: 1414259338

Splunk app for Stream automatically and permanently deletes the ephemeral stream when the server system time is greater than or equal to expirationDate.