Use Streams Config UI

What is a stream

A "stream" is a grouping of network event data that:

• has a network protocol;
• has protocol attributes;
• can be raw events or aggregate events;
• can be customized (cloned) to extract more granular event data.

About the Streams Config UI

The Streams Config UI lets you configure stream event capture for a wide variety of supported protocols. You can select protocol attributes and filter data capture with a high degree of specificity so that you capture only the event data that you require for analysis, while minimizing the quantity of data that Splunk must index.

The Streams Config UI lets you:

• Select the wire data protocols that you want to capture.
• Use filters to to define granular stream capture.
• Create custom (cloned) streams.
• Aggregate event data over user-defined intervals.
• View, enable, clone and delete ephemeral streams (in correlation with supported Splunk apps).

For details on how the streamfwd binary receives configuration data from the Streams Config UI, see How streamfwd communicates with splunk_app_stream.

Enable or disable event capture

To enable or disable event capture for specific protocols:

1. In the Splunk App for Stream menu, click Streams Config.

The Streams Config UI opens.

2. Select Stream Type: Permanent.

3. In the Group by menu, select how to group the Streams view: by Category, Application, or Protocol.

4. In the Streams window, click on the name of the protocol that you want to capture.

The configuration page opens for that specific protocol.

5. Select the status Enabled or Disabled button.

The selected protocol is now enabled/disabled for event capture.

Note: HTTP protocol is enabled by default.

Select protocol attribute fields

You can choose the specific fields that you want to capture for each stream protocol.

1. Select the Enable checkbox to add that field to the list of protocol events.

2. Click Save.

Add filters

You can add filters to a Stream based on conditions that compare a Term with a specific value. For example, for the term "http_method," you could add a condition that http_method must "contain" the value GET. If all (or any) http_method matches this condition, the stream event data passes through.

1. Click the Add button next to the term for which you want to create the filter condition.

2. Select a Comparison type from the drop-down menu.

3. Enter the Value that defines the condition of the comparison.

4. Click Save Changes.

Your new filter is added to the Stream configuration file and appears in the list of filters for that Stream.

Note: Filter comparison values can be strings or numeric values.

Create custom streams

You can create custom streams that provide additional granularity for network data capture. You can also add additional filters to custom streams and aggregate custom stream data over specific time intervals.

Clone streams

To create a custom stream, you must clone an existing stream. When you clone a stream, the app produces an exact duplicate of the original stream, including all enabled fields and existing filters.

1. In the main menu, click Streams.

This opens the main Streams Config page.

2. Click Clone.

The Clone Stream dialog appears.

3. Enter a Name and Description for your stream. Click OK.

A new Streams Config page opens for the cloned stream.

'4. Click Enabled to enable capture for the cloned stream.

5. Click Save.

Your new custom stream appears in the list of streams on the main Streams Config page.

Aggregate events

Splunk App for Stream lets you aggregate events for custom (cloned) streams over specific time-intervals. The app groups events into aggregation buckets, with one bucket allocated for each unique collection of "Key" fields. At the end of the time-interval, the app emits an object that represents each bucket.

For example, if you choose to aggregate bytes_in over an interval of 60 seconds, the app aggregates the total number of bytes_in over a 60 second interval in a unique bucket.

Aggregate Types

Key: Fields that have aggregate type Key are used for grouping and are included in the stream objects.

Sum: The values of fields that have aggregate type Sum are added together for all events over the time-interval.

To aggregate events:

1. On the Streams Config page, click on the name of your custom stream.

The configuration page for that custom stream opens.

2. Select the Aggregated checkbox.

This changes the Stream Type to "agg_events" and disables all fields.

3. Enter the Time Interval (in seconds) for aggregated events.

4. Select the Enable checkbox for each field you want to aggregate.

5. Select Key to define the aggregate bucket. This gives you a separate bucket for each value of the term. For example, if you select Key for the field "protocol" the app creates a separate bucket for each Level 7 protocol name (http, ftp, etc.).

6. Select Sum to aggregate numeric values for a field.

If your aggregate event includes multiple Key fields, Splunk App for Stream looks for unique combinations of Key event values and creates a unique bucket for each combination.

For example, if you assign Key to the field "protocol" and you assign Sum to the field "bytes_in," the app creates a unique bucket for each protocol value (http, ftp, etc.), and sums the number of bytes_in over the defined interval for each bucket. If you also assign Key to the "src_ip" and "dest_ip" fields, the app creates a unique bucket for each permutation of protocol, source, and destination IP address.

Support for ephemeral streams

Ephemeral streams give users of external Splunk apps the ability to schedule stream capture for a user-defined period of time. This is useful if you want to capture and analyze a limited number of network events pertaining to specific network activity or transactions over a specific time interval.

While you must configure and schedule ephemeral stream capture from within the context of the external Splunk app, you can view and perform certain actions on existing ephemeral streams (such as enable/disable) inside the Streams Config UI.

How ephemeral streams work

Splunk apps that support ephemeral streams take advantage of the Stream REST API. Ephemeral streams are similar to normal streams, but ephemeral streams require two additional parameters, createDate and expirationDate, which specify stream creation and expiration, respectively, in epoch time. For example:

createDate: 1404259338
expirationDate: 1414259338

Splunk app for Stream automatically and permanently deletes the ephemeral stream when the server system time is greater than or equal to expirationDate.