Set up ingest-time log-to-metrics conversion in Splunk Web

You can set up ingest-time log-to-metrics conversion through Splunk Web. You might want log-to-metrics conversion to take place at ingest time if you want the Splunk platform to preserve the metric data points that result from the conversion in a specific metrics index.

Complete the following two tasks to set up log-to-metrics conversion at ingest time:

• Create a source type in the Log to Metrics category.
• Apply this source type to a log data input.

To use this functionality, your role must have the edit_metric_schema capability. If your role does not have it, and you need to set up ingest-time logs-to-metrics conversion through Splunk Web, contact your Splunk administrator.

Know your log data

Creation of a Log to Metrics source type requires you to have basic knowledge about the log data that you wish to convert into metric data points. You need to know the fields in your log data and the categories that those fields fit into.

Field category	Description
Measurement	A field that provides the numeric value for a specific metric. A single metric data point can contain mulitiple measurements.
Dimension	A field that provides additional metadata for a metric data point. The Splunk platform counts as dimensions any fields it extracts from a log event that you have not already identified as measurements or excluded fields. A single metric data point can contain multiple dimensions.
Excluded field	A field in a log event that does not appear in the metric data point generated from that event. High-cardinality fields that are unimportant for the purposes of metric data point collection are good candidates for excluding.

For example, say you have an event with a timestamp and the following five fields: max_kb, min_kb, server_model, group, and division. If you identify max_kb and min_kb as measurements, and you identify group and division as excluded fields, the Splunk platform generates one metric data point that has metric_name:max_kb and metric_name:min_kb as measurements and server_model as a dimension field.

Create a Log to Metrics source type

You can create a source type in the Log to Metrics category with the Source Types listing page in Settings.

Prerequisites

• See Convert event logs to metric data points
• See Manage source types in Getting Data In for a full overview of the Source Types listing page and the process for adding a new source type.

Steps

1. Select Settings > Source types to open the Source Types listing page.
2. Click New Source Type to open the Create Source Type dialog.
3. Enter a Name for your new source type.
4. (Optional) Enter a source type Description for your new source type. Select a different Destination app if necessary.
5. Select Category > Log to Metrics.
6. Select an appropriate Indexed Extractions value for your data.
   
   For example, if you are working with structured CSV- or JSON-formatted data, select csv or json, as appropriate. Use field_extraction if your data is technically unstructured but its events are strings of field-value pairs.
   
   If you select field_extraction the Splunk software automatically adds WRITE_META=true to the transforms.conf stanza for the field extraction. See How the Splunk software builds indexed fields in Getting Data In.
7. (Optional) Change the settings on the Event Breaks, Timestamp, and Advanced tabs as necessary for your log data.
8. Click on the Metrics tab to reveal the Log to Metrics source type settings.
   

   Text box label	Optional?	Description
   Measures	No	Enter one or more comma-separated names of numeric measurement fields from the event data associated with the selected source type. The Splunk platform transforms each listed field into a measurement with a metric_name:<metric_name>::<numeric_value> syntax and then puts those measurements into the finished metric data point. You can use the wildcard character (*) to match multiple numeric measurement fields in your event data. For example, if your events contain max_size_kb, min_size_kb, and current_size_kb, you can include *_size_kb in the set of dimension field names. This adds all three fields to the set of measures. Alternatively, if you want the Splunk platform to treat all numeric fields in your event data as measures, just enter _ALLNUMS_ in the Measures field. If you want the Splunk platform to treat all but some numeric fields in your event data as measures, enter _NUMS_EXCEPT_ in the Measures field. Follow it with a space and then a comma-separated list of numeric fields from your event data that you do not want to extract as measures. These fields are instead extracted as dimensions.
   Whitelist	Yes	Enter one or more comma-separated names of dimension fields you want to include in the metric data points generated from the log events associated with this source type. All other dimension fields are excluded. You might want to set up a small list of included fields if most of the fields in your event data are high-cardinality or are otherwise unnecessary for your metrics. Use the wildcard character (*) to match multiple dimension field values in your event data. For example, if your event data contains customer_id, employee_id, and consultant_id as dimensions and you want to include all of them, you can add *_id to the set of dimension field names. This adds all three dimensions to the include list.
   Blacklist	Yes	Enter one or more comma-separated names of dimension fields that you want to exclude from the metric data points generated from the log events associated with this source type. All other dimension fields are included. You might want to exclude high-cardinality dimension fields that are unnecessary for your metric collection. Use the wildcard character (*) to match multiple dimension field values in your event data.

9. Click Save.

Apply a Log to Metrics source type to the data from an uploaded file or directory

After you create a source type in the Log to Metrics category, you can use the Set Source Type step of the Add Data workflow to apply the source type to data inputs that specify a single file as a source of data. When you set Log to Metric category source types to such inputs, a Metrics drop-down tab appears in the left pane of the Set Source Type page. Use this tab to enter or update lists of measures and blacklist dimensions for the source type.

The Add Data workflow is documented in full detail in Getting Data In.

Prerequisites

• See Overview of log-to-metrics functionality.
• See Create a Log to Metrics source type.
• See Monitor files and directories with Splunk Web in Getting Data In to review the Add Data workflow for inputs that specify a single file as a source of data.
• See The Set Sourcetype page in Getting Data In for an overview of the Set Source Type step of the Add Data workflow.

Steps

1. Follow the Add Data workflow for uploading or monitoring a file or directory until you get to the Select Source Type page.
2. On the Select Source Type page, select Source type > Log to Metrics and choose an appropriate source type from the list.
   When you select a Log to Metrics source type, the right-hand preview panel does not populate with a preview of the metrics data. You can see a preview for other source types.
3. (Optional) Open the Event Breaks, Timestamp, and Advanced drop-down tabs and update their settings as necessary for your data input.
4. (Optional) Open the Metrics drop-down tab to enter or update field lists in the Measures and Blacklist text boxes. Measures requires at least one field.
   

   Text box label	Description
   Measures	Review the entry in this text box and update it if necessary. It can contain a comma-separated list of numeric measurement fields from the event data that matches the selected source type. The Splunk platform transforms each listed field into a measurement with a metric_name:<metric_name>::<numeric_value> syntax and then puts those measurements into the finished metric data point. You can use the wildcard character (*) to match multiple numeric measurement fields in your event data. For example, if your events contain max_size_kb, min_size_kb, and current_size_kb, you can include *_size_kb in the set of dimension field names. This adds all three fields to the set of measures. It can contain just the term _ALLNUMS_. This tells the Splunk platform to transform all of the numeric fields in your event data into measures. Or it can contain the term _NUMS_EXCEPT_ followed by a space and a comma-separated list of numeric measurement fields. This tells the Splunk platform to convert all numeric fields in your event data into measurements except for the listed fields, which are instead extracted as dimensions.
   Whitelist	This text box can contain a comma-separated list of dimension fields that you specifically want to include from the metric data points generated from the log events associated with this source type. All dimension fields not in this list are excluded. You might want to set up an include list if most of the fields in your event data are high-cardinality or otherwise unnecessary for your metrics. Then you can keep just those fields that matter to you and dismiss the rest. Use the wildcard character (*) to match multiple dimension field values in your event data. For example, if your event data contains the dimensions customer_id, employee_id, and consultant_id, and you have *_id in the Blacklist text box, those are the only three dimensions that are included in the metric data points that are generated from the logs to metrics conversion.
   Blacklist	This text box can contain a comma-separated list of dimension fields that you specifically want to exclude from the metric data points generated from the log events associated with this source type. All dimension fields not in this list are included. You might want to exclude high-cardinality dimension fields that are unnecessary for your metric collection. Use the wildcard character (*) to match multiple dimension field values in your event data.

5. Click Next to continue with the Add Data workflow for your data input.