Set up ingest-time log-to-metrics conversion in Splunk Web

You can set up ingest-time log-to-metrics conversion through Splunk Web. You might want log-to-metrics conversion to take place at ingest time if you want the Splunk platform to preserve the metric data points that result from the conversion in a specific metrics index.

Complete the following two tasks to set up log-to-metrics conversion at ingest time:

• Create a source type in the Log to Metrics category.
• Apply this source type to a log data input.

To use this functionality, your role must have the edit_metric_schema capability. If your role does not have it, and you need to set up ingest-time logs-to-metrics conversion through Splunk Web, contact your Splunk administrator.

Know your log data

Creation of a Log to Metrics source type requires you to have basic knowledge about the log data that you wish to convert into metric data points. You need to know the fields in your log data and the categories that those fields fit into.

Field category	Description
Measurement	A field whose numeric values become unique metric data points.
Dimension	A field that provides additional metadata for metric data points. The Splunk platform counts as dimensions any fields it extracts from a log event that you have not already identified as measurements or blacklist fields. All metric data points generated from an event share the dimension field-value pairs in that event.
Blacklisted field	A field in a log event that does not appear in the metric data points generated from that event. High-cardinality fields that are unimportant for the purposes of metric data point collection are good candidates for field blacklisting.

For example, say you have an event with a timestamp and the following five fields: max_kb, min_kb, server_model, group, and division. If you identify max_kb and min_kb as measurements, and you identify group and division as blacklist fields, the Splunk platform will generate two metric data points, one for each of the measurement fields. The metric data points will both share server_model as a dimension field.

Create a Log to Metrics source type

You can create a source type in the Log to Metrics category with the Source Types listing page in Settings.

Prerequisites

• See Convert event logs to metric data points
• See Manage source types in Getting Data In for a full overview of the Source Types listing page and the process for adding a new source type.

Steps

1. Select Settings > Source types to open the Source Types listing page.
2. Click New Source Type to open the Create Source Type dialog.
3. Enter a Name for your new source type.
4. (Optional) Enter a source type Description for your new source type. Select a different Destination app if necessary.
5. Select Category > Log to Metrics.
6. Select an appropriate Indexed Extractions value for your data.
   
   For example, if you are working with structured CSV- or JSON-formatted data, select csv or json, as appropriate. Use field_extraction if your data is technically unstructured but its events are strings of field-value pairs.
7. (Optional) Change the settings on the Event Breaks, Timestamp, and Advanced tabs as necessary for your log data.
8. Click on the Metrics tab to reveal the Log to Metrics source type settings.
   

   Text box label	Optional?	Description
   Measures	No	Enter one or more comma-separated names of numeric measurement fields.
   Blacklist	Yes	Enter one or more comma-separated names of dimension fields that you want to blacklist from the metric data points generated from the log events associated with this source type. You might want to blacklist high-cardinality dimension fields that are unnecessary for your metric collection.

9. Click Save.

Apply a Log to Metrics source type to the data from an uploaded file or directory

After you create a source type in the Log to Metrics category, you can use the Set Source Type step of the Add Data workflow to apply the source type to data inputs that specify a single file as a source of data. When you set Log to Metric category source types to such inputs, a Metrics drop-down tab appears in the left pane of the Set Source Type page. Use this tab to enter or update lists of measures and blacklist dimensions for the source type.

The Add Data workflow is documented in full detail in Getting Data In.

Prerequisites

• See Overview of log-to-metrics functionality.
• See Create a Log to Metrics source type.
• See Monitor files and directories with Splunk Web in Getting Data In to review the Add Data workflow for inputs that specify a single file as a source of data.
• See The Set Sourcetype page in Getting Data In for an overview of the Set Source Type step of the Add Data workflow.

Steps

1. Follow the Add Data workflow for uploading or monitoring a file or directory until you get to the Select Source Type page.
2. On the Select Source Type page, select Source type > Log to Metrics and choose an appropriate source type from the list.
   When you select a Log to Metrics source type, the right-hand preview panel does not populate with a preview of the metrics data. You can see a preview for other source types.
3. (Optional) Open the Event Breaks, Timestamp, and Advanced drop-down tabs and update their settings as necessary for your data input.
4. (Optional) Open the Metrics drop-down tab to enter or update field lists in the Measures and Blacklist text boxes. Measures requires at least one field.
   

   Text box label	Description
   Measures	Review the list of comma-separated names of numeric measurement fields in this text box and update it if necessary. A unique metric data point is created for each measurement field-value pair in a log event associated with this source type.
   Blacklist	This text box can contain a comma-separated list of dimension fields that you want to blacklist from the metric data points generated from the log events associated with this source type. You might want to blacklist high-cardinality dimension fields that are unnecessary for your metric collection.

5. Click Next to continue with the Add Data workflow for your data input.