Configure Streams

The Configure Streams UI lets you configure passive capture of network event data for a variety of protocols.

A grouping of network event data is called a "stream." You can use the Configure Streams UI to create new streams for any supported protocol. You can also apply unique filtering and aggregation rules to any stream, which helps you capture only the data you require for analysis and minimize the amount of data sent to your indexers.

Use the Configure Streams UI to:

• Create new streams.
• Select stream protocols and attributes (fields).
• Apply filters for granular event capture.
• Aggregate event data over user-defined intervals.
• Enable estimates of stream index volume.
• Monitor ephemeral (time-limited) streams.
• Manage existing streams.

For a list of supported protocols, see Supported protocols in the Splunk App for Stream "Installation and Configuration Manual.

For details on how the streamfwd binary gets configuration data from the Configure Streams UI, see How streamfwd communicates with splunk_app_stream in the Splunk App for Stream Installation and Configuration Manual.

Create new stream

The Configure Stream UI provides a wizard that walks you through the process of creating a new stream. In addition to selecting a protocol and fields, the wizard lets you enable aggregation and create filters for the stream.

To create a new stream:

1. In the Splunk App for Stream main menu, click Configure > Configure Streams. The Configure Streams UI opens.

2. Click New Stream. The Create New Stream wizard workflow appears.

3. Under Basic Info, select a Protocol for your new stream. For example, HTTP. Enter a Name and optionally enter a Description. Click Next.

4. (optional) Under Aggregation, click Yes, Every. Then enter a time in seconds, for example, 60 seconds. This is the interval over which data aggregation occurs. For more information, see Aggregate events on this page.

5. Click Next.

6. Under Fields, select the specific protocol attributes (fields) that you want to capture. For example, dest_ip, src_ip, bytes_in, bytes_out, and so on. If you have enabled aggregation for the stream, you can also optionally select an Aggregation type (Key or Sum) for any field. Click Next.

7. (optional) Under Filters, click Create New Filter. The Create New Filter modal appears. Select the Field for which you want to create the filter, for example, http_method. Next, select the Comparison type. Then, enter the Value. Click Create. Your new filter allows data to pass through based on the condition defined in the filter. For more information on filters, see Create new filter on this page.

8. Click Next.

9. Under Settings, select the Index to which you want to send captured data. Then choose the Status for the new stream: Enabled, Disabled, or Estimate". Click Next.

10. Select the checkbox for the specific forwarder groups to which you want to add this stream. Click Create Stream. This creates the new stream.

11. Click Done. This sends your new stream configuration to the streamfwd binary on Splunk_TA_stream where data capture occurs.

Your new stream appears in the Configure Streams UI.

After you create a new stream, you can apply additional stream capture rules from the Configure Streams UI, including the Top "N" Fields filter, which lets you limit data capture to a specified number of top fields. See "Aggregate events" on this page.

You can also define Content Extraction rules that let you define a subset of data to capture from a protocol string field. See Use Content Extraction on this page.

Set stream mode

To enable data capture for a particular stream, you must set the stream to the Enabled mode. You can set the stream mode in the Configure Stream UI at any time. Choose from the following three modes:

• Enabled: This mode enables stream data capture and indexing, and generates index volume stats by default.
• Disabled: This mode disables stream data capture and indexing.
• Estimate: This mode generates data index volume stats for any stream, without sending data to indexers. For more information, see Use Estimate mode" on this page.

To select a mode for a stream:

On the Configure Streams UI main page, select a mode for any stream: Enabled, Estimate, or Disabled.

Note: All pre-built streams are in Estimate mode by default.

Select protocol attributes (fields)

You can choose the specific protocol attributes (fields) that you want to capture for any stream.

1. In the Configure Streams UI, select the name of the protocol for which you want to select/deselect fields.

This opens the Protocol Events page for the selected protocol.

2. Select the Enable checkbox for the specific field(s) you want to add/subtract.

3. Click Save.

The field is added/removed from the stream configuration.

Create new filters

You can add filters to a stream that allow data to pass through based on conditions that you define. You define these conditions by setting up a comparison between a field name and a specific value.

For example, a filter might specify that "http_method" data contains the value "GET." If all (or any) http_method events match this condition, the stream event data passes through.

To create a new filter:

1. In the Configure Streams UI, click on the name of the protocol. For example, http. The protocol events page opens.

2. Find the field for which you want to create a filter. For example, bytes_in, then click Actions > Add to Filter. The Filters modal opens, showing a list of existing filters for that field.

3. Select a Comparison type from the drop-down menu. For example, "Less than."

4. Enter the Value that defines the condition of the comparison. For example, "25000."

5. For Match Filters, select All or Any. This applies to events that return multiple values for a field. If match is set to "All", then all values of the field must satisfy the condition for the filter to engage. If the match is set to "Any", then any value of the field that satisfies the condition causes the filter to engage.

5. Click OK.

The app adds the filter to the streamfwd.xml configuration file, and the filter now appears in the list of filters for that stream.

Note: Filter comparison values can be strings or numeric values.

Aggregate events

The Configure Streams UI lets you set up aggregation of stream events over specific time-intervals. The app groups events into aggregation buckets, with one bucket allocated for each unique collection of "Key" fields. At the end of the time-interval, the app emits an object that represents each bucket.

For example, if you choose to aggregate bytes_in over an interval of 60 seconds, the app aggregates the total number of bytes_in over a 60 second interval in a unique bucket.

You must set each field that you enable for aggregation to one of the following two aggregation types:

• Key: Fields that have aggregation type Key are used for grouping data into buckets.
• Sum: Fields that have aggregation type Sum are added together for all events over the time-interval.

To set up aggregation:

1. In the Configure Streams UI, click on the name of your stream.

The events page for that particular page opens.

2. Under Aggregation, click Yes, every, then enter a time in secs.

This enables Aggregation for the stream and determines the time interval over which data aggregation occurs.

3. (optional) Under Top Fields, click Yes, only index top, then enter a number. You can define top fields based on count, bytes_in, bytes_out, and time_taken.

4. Select the Enable checkbox for each field you want to aggregate.

5. Select an Aggregation Type (Key or Sum) for each enabled field.

• Select Key to define the field as an aggregation bucket. This gives you a separate bucket for each value of the field. For example, if you select Key for the field src_ip the app creates a separate bucket for each src_ip.

• Select Sum to aggregate numeric values for the field. For example, if you select Sum for bytes_in the app adds together the total number of bytes_in over the time_interval.

6. Click Save.

If an aggregate event includes multiple Key fields, Splunk App for Stream looks for unique combinations of Key event values and creates a unique bucket for each combination.

For example, if you assign Key to the field "src_ip" and you assign Sum to the field "bytes_in," the app creates a unique bucket for each src_ip value, and sums the number of bytes_in over the defined interval for each bucket. If you also assign Key to the "dest_ip" field, the app creates a unique bucket for each permutation of src_ip, dest_ip, and bytes_in.

Use Content Extraction

The Configure Streams UI lets you define content extraction rules that capture and index a subset of data from a protocol string field.

For example, some string fields, such as src_content dest_content, and cookies, contain long strings of data. In many cases, the entire string of data is not useful. You might however want to capture a smaller section of the data, such as a name, ID, account number and so on. Content extraction lets you capture the specific pieces of data that you require, without indexing extraneous data.

Create content extraction rules

Content extraction rules use a regular expression to extract sections of data from the original field. You can also specify a capturing group match, which outputs either the first value that matches the regular expression, or the "list" of all values that match the regular expression.

Each content extraction rule creates a new field that captures only that data specified by the rule. The original field is not modified.

1. In the Configure Streams UI, click on the stream that contains the field from which you want to extract content.

The events page opens for the particular stream.

2. Click the Actions menu for the particular field, then select Extract New Field.

The Content Extraction dialog appears.

Note: It is not necessary for the original field to be enabled. If you also want to index the original field, consider using a field transformation instead. For more information, see field transformations in the Knowledge Manager Manual.

3. Enter a name for the content extraction rule. For example, "readable_cookie."

4. Enter a description for the content extraction rule. For example, "Separate the cookies into name/value pairs in a readable format."

5. In the Extraction Rule field, enter the regular expression for the content that you want to extract. For example, we can use following regular expression to extract a name and value pair from the cookie field:

(.*?)\=(.*?;)

Note: Stream uses Boost Perl Regular Expression syntax.

6. Under Match:

• Select First to return only the first value that matches the regex.
• Select All to return the "list" of all values that match the regex.

7. In the Extraction Format plain text box, enter the format for the extraction. For example, enter $1, $2 to return the first and second values that match the regex.

This image shows the complete content extraction rule for our "readable_cookie" example:

8. Click Save.

The new field appears in the list of Fields on the events page for the particular stream.

Splunk search results for the "readable_cookie" field in our example should appear similar to this:

Note: The <MaxEventQueueSize> option in streamfwd.xml determines the maximum number of events that Splunk_TA_stream can queue for delivery to Splunk indexers. By default <MaxEventQueueSize> supports 10k events. To increase or decrease the maximum event queue size, modify the value of <MaxEventQueueSize> in streamfwd.xml. See Configure streamfwd.xml in the Splunk App for Stream Installation and Configuration Manual.

Use Estimate mode

When you set a stream to Estimate mode, the app generates index volume stats for the stream, without sending the actual data to your indexers. These index volume stats populate the Stream Estimate dashboard.

Use the Estimate mode and Stream Estimate dashboard to determine the amount of data that a particular stream will ingest. This can help you calculate your indexer requirements, fine-tune your stream capture configurations, and conserve indexer space.

Note: Splunk App for Stream also collects index volume stats for all streams in the Enabled mode.

To use Estimate mode:

In the Configure Stream UI, select Mode > Estimate for any stream.

The stream is now in Estimate mode and begins to populate the Stream Estimate dashboard with index volume stats for the specific stream. For more information, see "Stream Estimate" in the this manual.

Create custom (cloned) streams

The Configure Streams UI lets you clone existing streams to create new custom streams. You can use the clone stream feature to create new streams that have additional granularity.

When you clone a stream, the app produces an exact duplicate of the original stream, including all enabled fields and existing filters. You can then add additional filters and capture rules to the stream.

To create a custom (cloned) stream:

1. In the Configure Streams UI, click on the name of the stream you want to clone.

2. The events page for the particular stream opens.

3. Click Clone.

The Clone Stream dialog appears.

3. Enter a Name and Description for the new stream. Click OK.

The new stream appears in the list of streams in the Configure Streams UI.

4. Click Enabled to enable capture for the cloned stream.

5. Click Save.

About built-in streams

Splunk App for Stream includes several built-in streams. These streams are provided as examples that you can use as a starting point to create new streams.

The data that built-in streams capture is used to populate Splunk App for Stream Informational dashboards. For more information, see "Informational Dashboards" in this manual.

All built-in streams begin with "Splunk_" in their name. You can view and clone built-in streams in the Configure Streams UI.

Support for ephemeral streams

Ephemeral streams give users of external Splunk apps the ability to schedule stream capture for a user-defined period of time. This is useful if you want to capture and analyze a limited number of network events pertaining to specific network activity or transactions over a specific time interval.

While you must configure and schedule ephemeral stream capture from within the context of the external Splunk app, you can view and perform certain actions on existing ephemeral streams (such as enable/disable) inside the Configure Stream UI.

How ephemeral streams work

Splunk apps that support ephemeral streams take advantage of the Stream REST API. Ephemeral streams are similar to normal streams, but ephemeral streams require two additional parameters, createDate and expirationDate, which specify the stream start and stop times, respectively, in epoch time. For example:

createDate: 1404259338
expirationDate: 1414259338

Splunk app for Stream automatically and permanently deletes the ephemeral stream when the server system time is greater than or equal to expirationDate.

To view ephemeral streams:

In the Configure Streams UI, click Ephemeral Streams. This opens the Stream Buckets page, which displays a list of your existing ephemeral streams.