Configure metadata streams

The Configure Streams UI provides a workflow wizard that walks you through the process of creating a new stream. When you create a new metadata stream you select a protocol and fields, apply aggregation, create filters for the stream.

About built-in metadata streams

Splunk Stream includes several built-in metadata streams as examples. You can use these built-in streams as a starting point for creating new metadata streams. All built-in streams begin with "Splunk_" in their name. You can view and clone built-in streams in the Configure Streams UI.

The data that built-in metadata streams capture populates the apps Informational dashboards. For more information, see Informational Dashboards in this manual.

Create a new metadata stream

1. In the Splunk App for Stream main menu, click Configure > Configure Streams. The Configure Streams UI opens.
2. Click New Stream > Metadata Stream.
3. In the Create New Stream workflow under Basic Info, select a Protocol for your new stream. For example, "http".
4. Enter a Name and optionally enter a Description. Click Next.
5. (optional) For Aggregation, click Yes, Every. Then enter a time in seconds, for example, 60 seconds. This is the interval over which data aggregation occurs. For more information, see Use Aggregation in this manual.
6. Click Next.
7. For Fields, select the specific protocol attributes (fields) that you want to capture. For example, dest_ip, src_ip, bytes_in, or bytes_out. If you have enabled aggregation for your stream, you can also optionally change the selected Aggregation type (Key or Aggregated) for any field. Click Next.
8. (optional) Under Filters, click Create New Filter.
   • Select the Field for which you want to create the filter, for example, "http_method."
   • Select the Comparison type.
   • Enter the Value.
   • Click Create. Your new filter allows data to pass through based on the condition defined in the filter.
9. Click Next.
10. Under Settings, select the Index to which you want to send captured data. Then choose the Status for the new stream: Enabled, Disabled, or Estimate.
11. Click Next.
12. Select the forwarder groups to which you want to add this stream.
13. Click Create Stream.
14. Click Done.

After you create a new metadata stream, you can modify the stream to apply additional stream capture rules, including aggregation methods and filters. You can also define content extraction rules to capture a subset of data from a protocol string. See Configure Streams to apply aggregation and Configure streams to use content extraction.

Set stream mode

To enable data capture for a stream, you must set the stream to Enabled mode. You can set the stream mode in the Configure Stream UI at any time. Choose from the following three modes:

• Enabled: Enabled mode starts stream data capture and indexing, and generates index volume stats by default.
• Estimate: Estimate mode generates data index volume stats only for any stream, without sending data to indexers.
• Disabled: Disabled mode stops stream data capture and indexing, and stops the collection of index volume stats.

Use Estimate mode

When you set a metadata stream to Estimate mode, the app generates index volume stats for your stream without sending the actual data to your indexers. These index volume stats populate the Stream Estimate dashboard.

Use the Estimate mode and Stream Estimate dashboard to determine the amount of data that a particular stream will ingest. This can help you calculate your indexer requirements, fine-tune your stream capture configurations, and conserve indexer space.

For more information, see Stream Estimate in this manual.

Splunk Stream also collects index volume stats for all streams in the Enabled mode. All pre-built streams are in Estimate mode by default.

Select protocol fields

Select the fields that you want to capture for a stream.

1. On the Configure Streams main page, select the name of the network protocol that you want to capture.
2. Select the Enable checkbox for the specific field(s) you want to add.
3. Click Save.

Create new filters

Add filters to fields in a stream that allow data to pass through based on conditions which you define. You define conditions by setting up a comparison, such as "Less than," "Equals," or "Contains," between a field name and a specific value. Filter values can be strings or numeric values.

For example, a filter can specify the condition that the "http_method" field contains the value "GET." If all (or any) http_method events match this condition, the stream event data passes through.

1. In the Configure Streams UI, click on the name of the stream.
2. Click the Filters tab.
3. Click Create New Filter.
4. In the Field drop-down menu, select the name of the field to apply to the filter.
5. In the Comparison menu, select the type of comparison you want to use for your filter.
6. Enter the Value that defines the condition of the comparison.
7. Click Save.
8. For Match Filters, select All or Any. This applies to events that return multiple values for a field. If Match Filters is set to "All", then all values for the field must satisfy the conditions for the filter to engage. If set to "Any", then any value of the field that satisfies the condition causes the filter to engage.
9. Click OK. The filter appears in the list of filters for that stream.