Set up ingest-time log-to-metrics conversion with configuration files

If you have access to the props.conf and transforms.conf files for your deployment, you can manually configure log-to-metrics transformations that are more sophisticated than the ones you can set up with Splunk Web. For example, you can design log-to-metrics transformations that can handle logs where not all of the events have the same sets of measurement and dimension fields.

To configure a log-to-metrics conversion, you need to add stanzas to your props.conf and transforms.conf files.

1. Specify the schema for log-to-metrics transformations in transforms.conf.
2. Configure log-to-metrics settings in props.conf.

For an overview of ingest-time conversion of logs to metric data points, see Convert event logs to metric data points.

Log-to-metric feature extensions that are not available in Splunk Web

When you manage log-to-metric processing through direct edits to configuration files, you can take advantage of optional feature extensions that are not yet available in Splunk Web.

Extended feature	In Splunk Web	Through configuration file edits	Setting
Sort events by the values of a shared field and target different log-to-metric conversion rules to each set of events.	The log-to-metric conversion rules are applied to all events that belong to the selected source type.	If all of the events belonging to the selected source type all share a metric_name field, you can devise log-to-metric rules for events with specific values of that field.	METRIC-SCHEMA-MEASURES-<unique_metric_name_prefix>, METRIC-SCHEMA-BLACKLIST-DIMS-<unique_metric_name_prefix>, and METRIC-SCHEMA-WHITELIST-DIMS-<unique_metric_name_prefix>.

Considerations for forwarders

When processing log-to-metrics conversions, the type of forwarder that you are using and the type of data that you are ingesting require specific indexer versions and locations for the transforms.conf and props.conf files with the log-to-metrics configurations.

The following configurations apply whether the data involved is structured or unstructured. Structured data includes formats like CSV and JSON. For more information, see Set up field extractions for the log data source.

Type of data	Forwarder version and type	Indexer version required	Location of log-to-metrics configuration files
Structured	Universal Forwarder with version 7.3.0 or later (but earlier than 8.x)	7.x or above	Universal Forwarder
Unstructured	Any Universal Forwarder version earlier than 8.x	7.3.x or above	Indexer
Structured	8.x Universal Forwarder	8.x	Universal Forwarder
Unstructured	8.x Universal Forwarder	8.x	Indexer
Structured	7.3.x Heavy Forwarder	7.x or above	Heavy Forwarder
Unstructured	7.3.x Heavy Forwarder	7.x or above	Heavy Forwarder
Structured	8.x Heavy Forwarder	8.x	Heavy Forwarder
Unstructured	8.x Heavy Forwarder	8.x	Heavy Forwarder

Specify the schema for log-to-metrics transformations in transforms.conf

Use configurations in the transforms.conf file to identify which events in a log contain metrics data points that you want to extract, and then specify how to extract the metrics from the log events.

1. Identify which events in a log contain metrics data points that you want to extract, and then apply the relevant settings in the configuration.
   • Apply log-to-metrics settings to all events in a log.
   • Apply log-to-metrics settings to specific events in a log.
2. Specify how to extract measures from log events.
3. Determine which event fields are transformed into metric dimensions.

Log-to-metrics metric schema settings reference

The metric schema settings determine how the log events associated with the stanza are transformed into metric data points. This table describes the syntax for the available settings when configuring log-to-metrics in transforms.conf:

Metric schema setting syntax	Description	Required?
METRIC-SCHEMA-MEASURES = (_ALLNUMS_ | (_NUMS_EXCEPT_ )? <field1>, <field2>,... )	Identifies how to extract numeric fields in events as measures in the metric data points that correspond to those events. You can identify the numeric fields that should be turned into measures, or you can process all numeric fields in an event as measures.	Yes
METRIC-SCHEMA-BLACKLIST-DIMS = <dimension_field1>, <dimension_field2>,...	Identifies excluded dimension fields. These are fields in your event data that cannot appear as dimensions in the metric data points that are generated from an event associated with the [metric-schema] stanza. You might want to set up a blacklist if some of the fields in your event data are high-cardinality dimension fields that are unnecessary for your metric collection.	No
METRIC-SCHEMA-WHITELIST-DIMS = <dimension_field1>,<dimension_field2>,...	Identifies whitelisted dimension fields. These are fields in your event data that must appear as dimensions in the metric data points that are generated from an event associated with the [metric-schema] stanza. You might want to set up a whitelist if most of the fields in your event data are high-cardinality or otherwise unnecessary for your metrics.	No

Apply log-to-metrics settings to all events in a log

Use the METRIC-SCHEMA-MEASURES setting to apply log-to-metrics processing to all events in a log. You can optionally use the METRIC-SCHEMA-BLACKLIST-DIMS and METRIC-SCHEMA-WHITELIST-DIMS settings to filter unnecessary dimension fields out of the resulting metric data points.

This configuration syntax is as follows:

[metric-schema:<unique_transforms_stanza_name]
METRIC-SCHEMA-MEASURES = (_ALLNUMS_ | (_NUMS_EXCEPT_ )? <field1>, <field2>,... )
METRIC-SCHEMA-BLACKLIST-DIMS = <dimension_field1>, <dimension_field2>,...
METRIC-SCHEMA-WHITELIST-DIMS = <dimension_field1>,<dimension_field2>,...

Replace (_ALLNUMS_ | (_NUMS_EXCEPT_ )? <field1>, <field2>,... ) with the specific setting you choose from Specify how to extract metrics from log events.

Apply log-to-metrics settings to specific events in a log

Use the METRIC-SCHEMA-MEASURES-<unique_metric_name_prefix> setting to apply log-to-metrics processing to specific events within a log. It will target a subset of events according to the value of a field shared by all of the events in the log.

Optionally use the METRIC-SCHEMA-BLACKLIST-DIMS-<unique_metric_name_prefix> and METRIC-SCHEMA-WHITELIST-DIMS-<unique_metric_name_prefix> parameters to filter unnecessary dimension fields out of the resulting metric data points.

This configuration syntax is as follows:

[metric-schema:<unique_transforms_stanza_name>]
METRIC-SCHEMA-MEASURES-<unique_metric_name_prefix> = (_ALLNUMS_ | (_NUMS_EXCEPT_ )? <field1>, <field2>,... )
METRIC-SCHEMA-BLACKLIST-DIMS-<unique_metric_name_prefix> = <dimension_field1>, <dimension_field2>,...
METRIC-SCHEMA-WHITELIST-DIMS-<unique_metric_name_prefix> = <dimension_field1>,<dimension_field2>,...

Replace (_ALLNUMS_ | (_NUMS_EXCEPT_ )? <field1>, <field2>,... ) with the specific setting you choose from Specify how to extract metrics from log events.

The <unique_metric_name_prefix> must match the value of a metric_name field that is shared by all of the events associated with the [metric-schema] stanza. The values of the metric_name field must correspond to the different event types present in the metric-schema stanza.

If a metric_name field is not already shared by your log events, there are ways to add it to your events:

• Create an index-time field extraction named metric_name.
• Use the INGEST_EVAL setting to add a metric_name field to the events at ingest time. For an example that shows you how to configure this, see Example of targeted log-to-metrics conversions.

When configured correctly, the METRIC-SCHEMA-MEASURES-<unique_metric_name_prefix> setting produces metric data points with measurements that follow this syntax: metric_name:<unique_metric_name_prefix>.<measure_field_name>=<numeric_measure_field_value>.

Always use the METRIC-SCHEMA-BLACKLIST-DIMS-<unique_metric_name_prefix> and METRIC-SCHEMA-WHITELIST-DIMS-<unique_metric_name_prefix> settings in conjunction with a corresponding METRIC-SCHEMA-MEASURES-<unique_metric_name_prefix> setting.

Specify how to extract measures from log events

There are several options to extract measures from log events:

• You can extract all numeric fields in events as measures.
• You can extract numeric fields with some exclusions as measures, or exclude specific fields from being extracted as measures.
• You can extract specific fields as measures, or whitelist specific fields to be extracted as measures.

These options are available whether you apply log-to-metrics settings to all events in a log or only to specific events in a log.

Method for extracting measures	Description	Syntax example	Fields with numeric and non-numeric values
Extract all numeric fields as measures.	Set up a [metric-schema] stanza using the _ALLNUMS_ setting.	[metric-schema:<unique_transforms_stanza_name] METRIC-SCHEMA-MEASURES = _ALLNUMS_	The _ALLNUMS_ setting extracts numeric values as measures for the field. Due to the non-numeric values, the same field is also used as a dimension field. If you want that field to be used only as a measure, exclude it as a dimension field. See Determine which event fields are transformed into metric dimensions.
Extract numeric fields with some exclusions as measures.	Set up a [metric-schema] stanza using the _NUMS_EXCEPT_ setting to define a blacklist of fields that you do not want extracted as measures. You must have a space between _NUMS_EXCEPT_ and the field name for the setting to function.	[metric-schema:<unique_transforms_stanza_name] METRIC-SCHEMA-MEASURES = _NUMS_EXCEPT_ <measure_field1>, <measure_field2>,...	The _NUMS_EXCEPT_ setting extracts the numeric values as measures for the field. If you want a field with both numeric and non-numeric fields to only be a dimension field, exclude it from being extracted as a measure using the _NUMS_EXCEPT_ setting.
Extract specific fields as measures.	In transforms.conf, set up a [metric-schema] stanza that identifies lists of fields that contain measurement values to extract only those fields as measures.	[metric-schema:<unique_transforms_stanza_name>] METRIC-SCHEMA-MEASURES = <measure_field1>, <measure_field2>,...	If you specify a field that has both numeric and non-numeric values with this setting, the numeric values are extracted as measures and the non-numeric values are ignored. The field is not used as a dimension field with the non-numeric values.

You can use the wildcard character to match multiple similar numeric fields in your data. For example, say your event data contains the following numeric fields max_size_kb, min_size_kb, and current_size_kb. You can set a <measure_field> value of *_size_kb to include all three of those numeric fields in the list of measures without listing each one separately.

Determine which event fields are transformed into metric dimensions

Any event field that the METRIC-SCHEMA-MEASURES setting does not identify as a measure can appear in the metric data point that is generated from that event as a dimension.

However, you can optionally use the METRIC-SCHEMA-BLACKLIST-DIMS and METRIC-SCHEMA-WHITELIST-DIMS settings to filter dimensions out of the metric data point.

• If you provide a list of event fields for METRIC-SCHEMA-BLACKLIST-DIMS, the search head transforms all unlisted non-measure fields into metric data point dimensions.
• If you provide a list of event fields for METRIC-SCHEMA-WHITELIST-DIMS, the fields in that list are the only fields that the search head transforms into metric data point dimensions. It ignores all other non-measure fields

The syntax for this configuration looks like this:

[metric-schema:<unique_transforms_stanza_name>]
METRIC-SCHEMA-MEASURES = <your_measures_setting>
METRIC-SCHEMA-BLACKLIST-DIMS = <dimension_field1>, <dimension_field2>,...
METRIC-SCHEMA-WHITELIST-DIMS = <dimension_field1>, <dimension_field2>,...

The search processor uses the following evaluation logic when a metric schema has fields defined for both METRIC-SCHEMA-BLACKLIST-DIMS and METRIC-SCHEMA-WHITELIST-DIMS:

• If a dimension is listed in the BLACKLIST, it won't be present in the resulting metric data point, even if it also appears in the WHITELIST.
• If a dimension is not listed in the WHITELIST, it won't be present in the resulting metric data point, even if it also does not appear in the BLACKLIST.

You can use the wildcard character to match multiple similar dimension fields in your data. For example, say your event data contains the following dimensions customer_id, employee_id, and consultant_id. You can set a <dimension_name> value of *_id to include all three of those dimensions in the dimension field list without listing each one separately.

If you have a field with both numeric and non-numeric values and you are using the _ALNUMS_ setting or the _NUMS_EXCEPT_ setting, the field is extracted as a measure by the setting and again as a dimension due to the non-numeric values. If you want that field to be used only as a measure, exclude it as a dimension field.

About extracting new fields from unstructured data or renaming fields

The Log to Metrics feature works by going over indexed fields in _meta to convert them to metric measurements or include or exclude them as dimensions. If you use structured data indexed field extraction methods (such as CSV or JSON extractions) you do not have to do extra setup beyond what we have provided here. However, if you are designing a more complex configuration that involves the extraction of fields from unstructured data, or the renaming of existing fields, you will need to make sure the fields are indexed. You can do this by setting WRITE_META=true in the transforms.conf stanza for the extraction.

See How Splunk software builds indexed fields, in Getting Data In.

Configure log-to-metrics settings in props.conf

After configuring the metrics schema for a source type in transforms.conf, finish configuring the log-to-metrics settings in props.conf. Configure log-to-metrics settings in the props.conf file:

1. Reference the metric schema from transforms.conf.
2. Set up field extractions for the log data source.

Reference the metric schema from transforms.conf

To associate the log-to-metrics schema with a specific log source type, reference the transforms.conf configuration in the stanza for the log source type in props.conf. Use the METRIC-SCHEMA-TRANSFORMS setting, which has the following syntax:

[ <sourcetype> ]
METRIC-SCHEMA-TRANSFORMS = <metric-schema:stanza_name>[,<metric-schema:stanza_name>]...

Type the names of the log-to-metrics transform stanzas in the <stanza_name> part of the METRIC-SCHEMA-TRANSFORMS setting.

Set up field extractions for the log data source

To use log-to-metrics configurations, you must design a configuration that extracts fields from your log data. The configuration that you use depends on whether the data is structured or unstructured.

If your log data is in a structured format like a CSV file or JSON, add the INDEXED_EXTRACTIONS setting to the props.conf stanza. See Extract fields from files with structured data in Getting Data In.

If your log data is technically unstructured but is organized into field-value pairs that can easily be extracted, add TRANSFORMS-<class>=field_extraction to the stanza. This references the [field_extraction] stanza in transforms.conf, which is included by default with the Splunk platform. The [field_extraction] stanza uses a simple regular expression to extract field-value pairs from log data.

Order of operations for log-to-metrics conversion settings

The Splunk platform processes all METRIC-SCHEMA-MEASURES-<unique_metric_name_prefix> and METRIC-SCHEMA-BLACKLIST-DIMS-<unique_metric_name_prefix> settings ahead of basic METRIC-SCHEMA-MEASURES and METRIC-SCHEMA-BLACKLIST-DIMS settings.

In other words, the Splunk platform processes all of the event-targeting log-to-metrics settings before it processes the event-agnostic log-to-metrics settings. This allows the latter group of settings to process remaining events that were not targeted by the <unique_metric_name_prefix> settings.

Example of targeted log-to-metrics conversions

Use targeted log-to-metrics conversions when one log source type contains multiple event schemas with different sets of measurements and dimension fields. The following event collection example contains two event schemas. The events share a group field, and the values of group identify the two event schemas.

_time	Event
08-05-2017 20:26:29.073 -0700	INFO Metrics - group=queue, location=sf, corp=splunk, name=udp_queue, max_size_kb=0, current_size_kb=0, current_size=0, largest_size=0, smallest_size=0
08-05-2017 20:26:29.073 -0700	INFO Metrics - group=queue, location=sf, corp=splunk, name=aggqueue, max_size_kb=1024, current_size_kb=1, current_size=5, largest_size=35, smallest_size=0
08-05-2017 20:26:29.073 -0700	INFO Metrics - group=queue, location=sf, corp=splunk, name=auditqueue, max_size_kb=500, current_size_kb=0, current_size=0, largest_size=1, smallest_size=0
08-05-2017 20:26:29.075 -0700	INFO Metrics - group=pipeline, name=indexerpipe, processor=indexin, cpu_seconds=0, executes=171, cumulative_hits=2214401
08-05-2017 20:26:29.075 -0700	INFO Metrics - group=pipeline, name=indexerpipe, processor=index_thruput, cpu_seconds=0, executes=171, cumulative_hits=2214401
08-05-2017 20:26:29.075 -0700	INFO Metrics - group=pipeline, name=indexerpipe, processor=indexandforward, cpu_seconds=0, executes=171, cumulative_hits=2214401

After examining these events, you decide that you need to define a set of configurations in transforms.conf and props.conf that perform the following tasks:

• Set TRANSFORMS-<class>=field_extraction to extract field-value pairs from the log lines at ingest time.
• Use INGEST_EVAL to add a metric_name field to every event with a group field at ingest time. The new metric_name fields get the same values as their corresponding group fields.
• Provide separate log-to-metrics settings for the metric_name=queue events and the metric_name=pipeline events. Extract all of the numeric fields from the metric_name=queue events as measures.
• Exclude the group, location, and corp fields from the dimensions for the metric_name=queue metric data points. Exclude the group field from the dimensions for the metric_name=pipeline events.
• Associate the log-to-metrics settings with events that have the metrics_log source type.

Those configurations look as follows:

transforms.conf

[eval_pipeline]
INGEST_EVAL = metric_name=group

[metric-schema:extract_metrics]
METRIC-SCHEMA-MEASURES-queue=_ALLNUMS_
METRIC-SCHEMA-BLACKLIST-DIMS-queue=group,location,corp
METRIC-SCHEMA-MEASURES-pipeline=cpu_seconds,executes,cumulative_hits
METRIC-SCHEMA-BLACKLIST-DIMS-pipeline=group

props.conf

[metrics_log]
TRANSFORMS-fieldvalue=field_extraction
TRANSFORMS-metricslog=eval_pipeline
METRIC-SCHEMA-TRANSFORMS=metric-schema:extract_metrics

The metric data points created by these configurations look like the following examples:

_time	measures	dimensions
08-05-2017 20:26:29.073 -0700	metric_name:queue.max_size_kb=0, metric_name:queue.current_size_kb=0, metric_name:queue.current_size=0, metric_name:queue.largest_size=0, metric_name:queue.smallest_size=0	name=udp_queue
08-05-2017 20:26:29.073 -0700	metric_name:queue.max_size_kb=1024, metric_name:queue.current_size_kb=1, metric_name:queue.current_size=5, metric_name:queue.largest_size=35, metric_name:queue.smallest_size=0	name=aggqueue
08-05-2017 20:26:29.073 -0700	metric_name:queue.max_size_kb=500, metric_name:queue.current_size_kb=0, metric_name:queue.current_size=0, metric_name:queue.largest_size=1, metric_name:queue.smallest_size=0	name=auditqueue
08-05-2017 20:26:29.075 -0700	metric_name:pipeline.cpu_seconds=0, metric_name:pipeline.executes=171, metric_name:pipeline.cumulative_hits=2214401	name=indexerpipe, processor=indexin
08-05-2017 20:26:29.075 -0700	metric_name:pipeline.cpu_seconds=0, metric_name:pipeline.executes=171, metric_name:pipeline.cumulative_hits=2214401,	name=indexerpipe, processor=index_thruput
08-05-2017 20:26:29.075 -0700	metric_name:pipeline.cpu_seconds=0, metric_name:pipeline.executes=171, metric_name:pipeline.cumulative_hits=2214401,	name=indexerpipe, processor=indexandforward