Set up ingest-time log-to-metrics conversion with configuration files

If you have access to the props.conf and transforms.conf files for your deployment, you can manually configure log-to-metrics transformations that are more sophisticated than the ones you can set up with Splunk Web. For example, you can design log-to-metrics transformations that can handle logs where not all of the events have the same sets of measurement and dimension fields.

To configure a log-to-metrics conversion, you need to add stanzas to your props.conf and transforms.conf files.

1. Specify the schema for log-to-metrics transformations in transforms.conf.
2. Configure log-to-metrics settings in props.conf.

For an overview of ingest-time conversion of logs to metric data points, see Overview of log-to-metrics conversion.

Log-to-metrics feature extensions that are unavailable in Splunk Web

When you manage log-to-metric processing through direct edits to configuration files, you can take advantage of optional feature extensions that are not yet available in Splunk Web.

Extended feature	In Splunk Web	Through configuration file edits	Setting
Automatically process all numeric fields as measures	You must list the fields that are measures in the Measures text box	Instead of listing fields for METRIC-SCHEMA-MEASURES, optionally use the _ALLNUMS_ argument, which causes the search head to process all numeric fields in an event as a measure.	_ALLNUMS_
Automatically process all numeric fields except a certain subset of fields as measures	You must list the fields that are measures in the Measures text box	Instead of just listing fields for METRIC-SCHEMA-MEASURES, optionally use the _NUMS_EXCEPT_ argument in conjunction with a list of numeric fields that should not be processed as a measure. This causes the search head to process all numeric fields as measures except the fields in the list.	_NUMS_EXCEPT_
Whitelist filtering for dimensions	You can only filter dimensions by providing a dimension blacklist. This is a set of dimensions that must be excluded from the metric data points generated by the log-to-metrics configuration.	Alternatively, you can filter out dimensions with a dimension whitelist. This is a set of dimensions that must be included in the metric data points generated by the log-to-metrics configuration. All dimensions not in the whitelist are excluded from the metric data points generated by the log-to-metrics configuration.	METRIC-SCHEMA-WHITELIST-DIMS

Considerations for forwarders

When processing log-to-metrics conversions, the type of forwarder that you are using and the type of data that you are ingesting require specific indexer versions and locations for the transforms.conf and props.conf files with the log-to-metrics configurations.

Structured data includes formats like CSV and JSON. For more information, see Set up field extractions for the log data source.

Forwarder version and type	Type of data	Indexer version required	Location of log-to-metrics configuration files
7.3x Universal Forwarder	Structured	7.x	Universal Forwarder
Any Universal Forwarder version	Unstructured	7.3.x	Indexer
7.3.x Heavy Forwarder	Structured	7.x	Heavy Forwarder
7.3.x Heavy Forwarder	Unstructured	7.x	Heavy Forwarder

Specify the schema for log-to-metrics transformations in transforms.conf

Use configurations in the transforms.conf file to identify which events in a log contain metrics data points that you want to extract, and then specify how to extract the metrics from the log events.

1. Identify which events in a log contain metrics data points that you want to extract, and then apply the relevant settings in the configuration.
   • Apply log-to-metrics settings to all events in a log.
   • Apply log-to-metrics settings to specific events in a log.
2. Specify how to extract measures from log events.
3. Blacklist dimension fields for metric transformations.

Log-to-metrics metric schema settings reference

The metric schema settings determine how each of the log events associated with the stanza is transformed into multiple metric data points. This table describes the syntax for the available settings when configuring log-to-metrics in transforms.conf:

Metric schema setting syntax	Description	Required?
METRIC-SCHEMA-MEASURES = (_ALLNUMS_ | (_NUMS_EXCEPT_ )? <field1>, <field2>,... )	Identifies how to extract fields as measures. The Splunk platform generates a separate metric data point for each measurement field-value pair in an event associated with the [metric-schema] stanza. When the metric data point is generated from the measurement field-value pair, the measurement field is used as the value for the metric_name field, and the measurement value is used as the value for the _value field.	Yes
METRIC-SCHEMA-BLACKLIST-DIMS = <dimension_field1>, <dimension_field2>,...	Identifies blacklisted dimension fields. These are fields in your event data that cannot appear as dimensions in the metric data points that are generated from an event associated with the [metric-schema] stanza. You might want to blacklist high-cardinality dimension fields that are unnecessary for your metric collection.	No
METRIC-SCHEMA-WHITELIST-DIMS = <dimension_field1>,<dimension_field2>,...	Identifies whitelisted dimension fields. These are fields in your event data that must appear as dimensions in the metric data points that are generated from an event associated with the [metric-schema] stanza. You might want to set up a whitelist if most of the fields in your event data are high-cardinality or otherwise unnecessary for your metrics.	No

Apply log-to-metrics settings to all events in a log

Use the METRIC-SCHEMA-MEASURES setting to apply log-to-metrics processing to all events in a log. You can optionally use the METRIC-SCHEMA-BLACKLIST-DIMS and METRIC-SCHEMA-WHITELIST-DIMS settings to filter unnecessary dimension fields out of the resulting metric data points.

This configuration syntax is as follows:

[metric-schema:<unique_transforms_stanza_name]
METRIC-SCHEMA-MEASURES = (_ALLNUMS_ | (_NUMS_EXCEPT_ )? <field1>, <field2>,... )
METRIC-SCHEMA-BLACKLIST-DIMS = <dimension_field1>, <dimension_field2>,...
METRIC-SCHEMA-WHITELIST-DIMS = <dimension_field1>,<dimension_field2>,...

Replace (_ALLNUMS_ | (_NUMS_EXCEPT_ )? <field1>, <field2>,... ) with the specific setting you choose from Specify how to extract metrics from log events.

Apply log-to-metrics settings to specific events in a log

Use the METRIC-SCHEMA-MEASURES-<unique_metric_name_prefix> setting to apply log-to-metrics processing to specific events within a log. It will target groups events according to the value of a field shared by all of the events in the log.

Optionally use the METRIC-SCHEMA-BLACKLIST-DIMS-<unique_metric_name_prefix> and METRIC-SCHEMA-WHITELIST-DIMS-<unique_metric_name_prefix> parameters to filter unnecessary dimension fields out of the resulting metric data points.

This configuration syntax is as follows:

[metric-schema:<unique_transforms_stanza_name>]
METRIC-SCHEMA-MEASURES-<unique_metric_name_prefix> = (_ALLNUMS_ | (_NUMS_EXCEPT_ )? <field1>, <field2>,... )
METRIC-SCHEMA-BLACKLIST-DIMS-<unique_metric_name_prefix> = <dimension_field1>, <dimension_field2>,...
METRIC-SCHEMA-WHITELIST-DIMS-<unique_metric_name_prefix> = <dimension_field1>,<dimension_field2>,...

Replace (_ALLNUMS_ | (_NUMS_EXCEPT_ )? <field1>, <field2>,... ) with the specific setting you choose from Specify how to extract metrics from log events.

The <unique_metric_name_prefix> must match the value of a metric_name field that is shared by all of the events associated with the [metric-schema] stanza. The values of the metric_name field must correspond to the different event types present in the metric-schema stanza.

If a metric_name field is not already shared by your log events, there are ways to add it to your events:

• Create an index-time field extraction named metric_name.
• Use the INGEST_EVAL setting to add a metric_name field to the events at ingest time. For an example that shows you how to configure this, see Example of targeted log-to-metrics conversions.

When configured correctly, the METRIC-SCHEMA-MEASURES-<unique_metric_name_prefix> setting produces metric data points with metric_name values that follow this syntax: <unique_metric_name_prefix>.<measure_field_name>.

Always use the METRIC-SCHEMA-BLACKLIST-DIMS-<unique_metric_name_prefix> and METRIC-SCHEMA-WHITELIST-DIMS-<unique_metric_name_prefix> settings in conjunction with a corresponding METRIC-SCHEMA-MEASURES-<unique_metric_name_prefix> setting.

Specify how to extract measures from log events

There are several options to extract measures from log events:

• You can extract all numeric fields in events as measures.
• You can extract numeric fields with some exclusions as measures, or blacklist specific fields from being extracted as measures.
• You can extract specific fields as measures, or whitelist specific fields to be extracted as measures.

These options are available whether you apply log-to-metrics settings to all events in a log or only to specific events in a log.

Method for extracting measures	Description	Syntax example	Fields with numeric and non-numeric values
Extract all numeric fields as measures.	Set up a [metric-schema] stanza using the _ALLNUMS_ setting.	[metric-schema:<unique_transforms_stanza_name] METRIC-SCHEMA-MEASURES = _ALLNUMS_	The _ALLNUMS_ setting extracts numeric values as measures for the field. Due to the non-numeric values, the same field is also used as a dimension field. If you want that field to be used only as a measure, blacklist it as a dimension field. See Blacklist dimension fields for metric transformations.
Extract numeric fields with some exclusions as measures.	Set up a [metric-schema] stanza using the _NUMS_EXCEPT_ setting to define a blacklist of fields that you do not want extracted as measures. You must have a space between _NUMS_EXCEPT_ and the field name for the setting to function.	[metric-schema:<unique_transforms_stanza_name] METRIC-SCHEMA-MEASURES = _NUMS_EXCEPT_ <measure_field1>, <measure_field2>,...	The _NUMS_EXCEPT_ setting extracts the numeric values as measures for the field. If you want a field with both numeric and non-numeric fields to only be a dimension field, exclude it from being extracted as a measure using the _NUMS_EXCEPT_ setting.
Extract specific fields as measures.	In transforms.conf, set up a [metric-schema] stanza that identifies lists of fields that contain measurement values to extract only those fields as measures.	[metric-schema:<unique_transforms_stanza_name>] METRIC-SCHEMA-MEASURES = <measure_field1>, <measure_field2>,...	If you specify a field that has both numeric and non-numeric values with this setting, the numeric values are extracted as measures and the non-numeric values are ignored. The field is not used as a dimension field with the non-numeric values.

Determine which event fields are transformed into metric dimensions

Any event field that the METRIC-SCHEMA-MEASURES setting does not identify as a measure can appear in the metric data points that are generated from that event as a dimension. A dimension field/value pair in an event is shared by all of the metric data points that are generated from that event.

However, you can optionally use the METRIC-SCHEMA-BLACKLIST-DIMS and METRIC-SCHEMA-WHITELIST-DIMS settings to filter dimensions out of the final set of metric data points.

• If you provide a list of event fields for METRIC-SCHEMA-BLACKLIST-DIMS, the search head transforms all non-measure fields that are not in that list into metric data point dimensions.
• If you provide a list of event fields for METRIC-SCHEMA-WHITELIST-DIMS, the fields in that list are the only fields that the search head transforms into metric data point dimensions. It ignores all other non-measure fields

The syntax for this configuration looks like this:

[metric-schema:<unique_transforms_stanza_name>]
METRIC-SCHEMA-MEASURES = <your_measures_setting>
METRIC-SCHEMA-BLACKLIST-DIMS = <dimension_field1>, <dimension_field2>,...
METRIC-SCHEMA-WHITELIST-DIMS = <dimension_field1>, <dimension_field2>,...

The search processor uses the following evaluation logic when a metric schema has fields defined for both METRIC-SCHEMA-BLACKLIST-DIMS and METRIC-SCHEMA-WHITELIST-DIMS:

• If a dimension is listed in the BLACKLIST, it won't be present in the resulting metric data points, even if it also appears in the WHITELIST.
• If a dimension is not listed in the WHITELIST, it won't be present in the resulting metric data points, even if it also does not appear in the BLACKLIST.

You can use the wildcard character to match multiple similar dimension fields in your data. For example, say your event data contains the following dimensions customer_id, employee_id, and consultant_id. You can set a <dimension_name> value of *_id to include all three of those dimensions in the dimension field list without listing each one separately.

If you have a field with both numeric and non-numeric values and you are using the _ALNUMS_ setting, the field is extracted as a measure by the _ALLNUMS_ setting and as a dimension due to the non-numeric values. If you want that field to be used only as a measure, blacklist it as a dimension field.

Configure log-to-metrics settings in props.conf

After configuring the metrics schema for a source type in transforms.conf, finish configuring the log-to-metrics settings in props.conf. Configure log-to-metrics settings in the props.conf file:

1. Reference the metric schema from transforms.conf.
2. Set up field extractions for the log data source.

Reference the metric schema from transforms.conf

To associate the log-to-metrics schema with a specific log source type, reference the transforms.conf configuration in the stanza for the log source type in props.conf. Use the METRIC-SCHEMA-TRANSFORMS setting, which has the following syntax:

[ <sourcetype> ]
METRIC-SCHEMA-TRANSFORMS = <metric-schema:stanza_name>[,<metric-schema:stanza_name>]...

Type the names of the log-to-metrics transform stanzas in the <stanza_name> part of the METRIC-SCHEMA-TRANSFORMS setting.

Set up field extractions for the log data source

To use log-to-metrics configurations, you must design a configuration that extracts fields from your log data. The configuration that you use depends on whether the data is structured or unstructured.

If your log data is in a structured format like a CSV file or JSON, add the INDEXED_EXTRACTIONS setting to the props.conf stanza. See Extract fields from files with structured data in Getting Data In.

If your log data is technically unstructured but is organized into field-value pairs that can easily be extracted, add TRANSFORMS-<class>=field_extraction to the stanza. This references the [field_extraction] stanza in transforms.conf, which is included by default with the Splunk platform. The [field_extraction] stanza uses a simple regular expression to extract field-value pairs from log data.

Order of operations for log-to-metrics conversion settings

The Splunk platform processes all METRIC-SCHEMA-MEASURES-<unique_metric_name_prefix> and METRIC-SCHEMA-BLACKLIST-DIMS-<unique_metric_name_prefix> settings ahead of basic METRIC-SCHEMA-MEASURES and METRIC-SCHEMA-BLACKLIST-DIMS settings.

In other words, the Splunk platform processes all of the event-targeting log-to-metrics settings before it processes the event-agnostic log-to-metrics settings. This allows the latter group of settings to process remaining events that were not targeted by the <unique_metric_name_prefix> settings.

Example of targeted log-to-metrics conversions

Use targeted log-to-metrics conversions when one log source type contains multiple event schemas with different sets of measurements and dimension fields. The following event collection example contains two event schemas. The events share a group field, and the values of group identify the two event schemas.

_time	Event
08-05-2017 20:26:29.073 -0700	INFO Metrics - group=queue, location=sf, corp=splunk, name=udp_queue, max_size_kb=0, current_size_kb=0, current_size=0, largest_size=0, smallest_size=0
08-05-2017 20:26:29.073 -0700	INFO Metrics - group=queue, location=sf, corp=splunk, name=aggqueue, max_size_kb=1024, current_size_kb=1, current_size=5, largest_size=35, smallest_size=0
08-05-2017 20:26:29.073 -0700	INFO Metrics - group=queue, location=sf, corp=splunk, name=auditqueue, max_size_kb=500, current_size_kb=0, current_size=0, largest_size=1, smallest_size=0
08-05-2017 20:26:29.075 -0700	INFO Metrics - group=pipeline, name=indexerpipe, processor=indexin, cpu_seconds=0, executes=171, cumulative_hits=2214401
08-05-2017 20:26:29.075 -0700	INFO Metrics - group=pipeline, name=indexerpipe, processor=index_thruput, cpu_seconds=0, executes=171, cumulative_hits=2214401
08-05-2017 20:26:29.075 -0700	INFO Metrics - group=pipeline, name=indexerpipe, processor=indexandforward, cpu_seconds=0, executes=171, cumulative_hits=2214401

After examining these events, you decide that you need to define a set of configurations in transforms.conf and props.conf that perform the following tasks:

• Set TRANSFORMS-<class>=field_extraction to extract field-value pairs from the log lines at ingest time.
• Use INGEST_EVAL to add a metric_name field to every event with a group field at ingest time. The new metric_name fields get the same values as their corresponding group fields.
• Provide separate log-to-metrics settings for the metric_name=queue events and the metric_name=pipeline events. Extract all of the numeric fields from the metric_name=queue events as measures.
• Blacklist the group, location, and corp fields from the dimensions for the metric_name=queue metric data points. Blacklist the group field from the dimensions for the metric_name=pipeline events.
• Associate the log-to-metrics settings with events that have the metrics_log source type.

Those configurations look as follows:

transforms.conf

[eval_pipeline]
INGEST_EVAL = metric_name=group

[metric-schema:extract_metrics]
METRIC-SCHEMA-MEASURES-queue=_ALLNUMS_
METRIC-SCHEMA-BLACKLIST-DIMS-queue=group,location,corp
METRIC-SCHEMA-MEASURES-pipeline=cpu_seconds,executes,cumulative_hits
METRIC-SCHEMA-BLACKLIST-DIMS-pipeline=group

props.conf

[metrics_log]
TRANSFORMS-fieldvalue=field_extraction
TRANSFORMS-metricslog=eval_pipeline
METRIC-SCHEMA-TRANSFORMS=metric-schema:extract_metrics

The metric data points created by these configurations look like the following examples:

_time	metric_name	_value	name	processor
08-05-2017 20:26:29.073 -0700	queue.max_size_kb	1024	aggqueue
08-05-2017 20:26:29.073 -0700	queue.current_size_kb	1	aggqueue
08-05-2017 20:26:29.073 -0700	queue.current_size	5	aggqueue
08-05-2017 20:26:29.073 -0700	queue.largest_size	35	aggqueue
08-05-2017 20:26:29.073 -0700	queue.smallest_size	0	aggqueue
08-05-2017 20:26:29.075 -0700	pipeline.cpu_seconds	0	indexerpipe	indexin
08-05-2017 20:26:29.075 -0700	pipeline.executes	171	indexerpipe	indexin
08-05-2017 20:26:29.075 -0700	pipeline.cumulative_hits	2214401	indexerpipe	indexin