Set up ingest-time log-to-metrics conversion with configuration files

If you have access to the props.conf and transforms.conf files for your deployment, you can manually configure log-to-metrics transformations that are more sophisticated than the ones you can set up with Splunk Web. For example, you can design log-to-metrics transformations that can handle logs where not all of the events have the same sets of measurement and dimension fields.

To configure a log-to-metrics conversion, you need to add stanzas to your props.conf and transforms.conf files.

1. Specify the schema for log-to-metrics transformations in transforms.conf.
2. Configure log-to-metrics settings in props.conf.

For an overview of ingest-time conversion of logs to metric data points, see Overview of log-to-metrics conversion.

Considerations for forwarders

When processing log-to-metrics conversions, the type of forwarder that you are using and the type of data that you are ingesting require specific indexer versions and locations for the transforms.conf and props.conf files with the log-to-metrics configurations.

Structured data includes formats like CSV and JSON. For more information, see Set up field extractions for the log data source.

Forwarder version and type	Type of data	Indexer version required	Location of log-to-metrics configuration files
7.3x Universal Forwarder	Structured	7.x	Universal Forwarder
Any Universal Forwarder version	Unstructured	7.3.x	Indexer
7.3.x Heavy Forwarder	Structured	7.x	Heavy Forwarder
7.3.x Heavy Forwarder	Unstructured	7.x	Heavy Forwarder

Specify the schema for log-to-metrics transformations in transforms.conf

Use configurations in the transforms.conf file to identify which events in a log contain metrics data points that you want to extract, and then specify how to extract the metrics from the log events.

1. Identify which events in a log contain metrics data points that you want to extract, and then apply the relevant settings in the configuration.
   • Apply log-to-metrics settings to all events in a log.
   • Apply log-to-metrics settings to specific events in a log.
2. Specify how to extract measures from log events.
3. Blacklist dimension fields for metric transformations.

Log-to-metrics metric schema settings reference

The metric schema settings determine how each of the log events associated with the stanza is transformed into multiple metric data points. This table describes the syntax for the available settings when configuring log-to-metrics in transforms.conf:

Metric schema setting syntax	Description	Required?
METRIC-SCHEMA-MEASURES = (_ALLNUMS_ | (_NUMS_EXCEPT_ )? <field1>, <field2>,... )	Identifies how to extract fields as measures. The Splunk platform generates a separate metric data point for each measurement field-value pair in an event associated with the [metric-schema] stanza. When the metric data point is generated from the measurement field-value pair, the measurement field is used as the value for the metric_name field, and the measurement value is used as the value for the _value field.	Yes
METRIC-SCHEMA-BLACKLIST-DIMS = <dimension_field1>, <dimension_field2>,...	Provides a list of blacklist dimension fields. These are fields that you do not want to appear as dimensions in the metric data points that are generated from an event associated with the [metric-schema] stanza. You might want to blacklist high-cardinality dimension fields that are unnecessary for your metric collection.	No

All fields that are not identified as measure fields for METRIC-SCHEMA-MEASURES or blacklist dimension fields for METRIC-SCHEMA-BLACKLIST-DIMS appear in the metric data points as dimensions. A dimension field-value pair in an event is shared by all of the metric data fields generated from that event.

Apply log-to-metrics settings to all events in a log

To apply log-to-metrics settings to all events in a log, use the METRIC-SCHEMA-MEASURES and METRIC-SCHEMA-BLACKLIST-DIMS parameters.

This configuration syntax is as follows:

[metric-schema:<unique_transforms_stanza_name]
METRIC-SCHEMA-MEASURES = (_ALLNUMS_ | (_NUMS_EXCEPT_ )? <field1>, <field2>,... )
METRIC-SCHEMA-BLACKLIST-DIMS = <dimension_field1>, <dimension_field2>,...

Replace (_ALLNUMS_ | (_NUMS_EXCEPT_ )? <field1>, <field2>,... ) with the specific setting you choose from Specify how to extract metrics from log events.

Apply log-to-metrics settings to specific events in a log

You can also extract metrics from logs that target specific groups of log events according to the value of a field shared by all events in that log.

This configuration syntax is as follows:

[metric-schema:<unique_transforms_stanza_name>]
METRIC-SCHEMA-MEASURES-<unique_metric_name_prefix> = (_ALLNUMS_ | (_NUMS_EXCEPT_ )? <field1>, <field2>,... )
METRIC-SCHEMA-BLACKLIST-DIMS-<unique_metric_name_prefix> = <dimension_field1>, <dimension_field2>,...

Replace (_ALLNUMS_ | (_NUMS_EXCEPT_ )? <field1>, <field2>,... ) with the specific setting you choose from Specify how to extract metrics from log events.

The <unique_metric_name_prefix> must match the value of a metric_name field that is shared by all of the events associated with the [metric-schema] stanza. The values of the metric_name field must correspond to the different event types present in the metric-schema stanza.

If a metric_name field is not already shared by your log events, there are ways to add it to your events:

• Create an index-time field extraction named metric_name.
• Use the INGEST_EVAL setting to add a metric_name field to the events at ingest time. For an example that describes how to configure this, see Example of targeted log-to-metrics conversions.

When configured correctly, the METRIC-SCHEMA-MEASURES-<unique_metric_name_prefix> setting produces metric data points with metric_name values that follow this syntax: <unique_metric_name_prefix>.<measure_field_name>.

Always use the METRIC-SCHEMA-BLACKLIST-DIMS-<unique_metric_name_prefix> setting in conjunction with a corresponding METRIC-SCHEMA-MEASURES-<unique_metric_name_prefix> setting.

Specify how to extract measures from log events

There are several options to extract measures from log events:

• You can extract all numeric fields in events as measures.
• You can extract numeric fields with some exclusions as measures, or blacklist specific fields from being extracted as measures.
• You can extract specific fields as measures, or whitelist specific fields to be extracted as measures.

These options are available whether you apply log-to-metrics settings for all events in a log or only specific events in a log.

Method for extracting measures	Description	Syntax example	Fields with numeric and non-numeric values
Extract all numeric fields as measures.	Set up a [metric-schema] stanza using the _ALLNUMS_ setting.	[metric-schema:<unique_transforms_stanza_name] METRIC-SCHEMA-MEASURES = _ALLNUMS_	The _ALLNUMS_ setting extracts numeric values as measures for the field. Due to the non-numeric values, the same field is also used as a dimension field. If you want that field to be used only as a measure, blacklist it as a dimension field. See Blacklist dimension fields for metric transformations.
Extract numeric fields with some exclusions as measures.	Set up a [metric-schema] stanza using the _NUMS_EXCEPT_ setting to define a blacklist of fields that you do not want extracted as measures. You must have a space between _NUMS_EXCEPT_ and the field name for the setting to function.	[metric-schema:<unique_transforms_stanza_name] METRIC-SCHEMA-MEASURES = _NUMS_EXCEPT_ <measure_field1>, <measure_field2>,...	The _NUMS_EXCEPT_ setting extracts the numeric values as measures for the field. If you want a field with both numeric and non-numeric fields to only be a dimension field, exclude it from being extracted as a measure using the _NUMS_EXCEPT_ setting.
Extract specific fields as measures.	In transforms.conf, set up a [metric-schema] stanza that identifies lists of fields that contain measurement values to extract only those fields as measures.	[metric-schema:<unique_transforms_stanza_name>] METRIC-SCHEMA-MEASURES = <measure_field1>, <measure_field2>,...	If you specify a field that has both numeric and non-numeric values with this setting, the numeric values are extracted as measures and the non-numeric values are ignored. The field is not used as a dimension field with the non-numeric values.

Blacklist dimension fields for metric transformations

Specify fields to exclude as dimensions in the metric data points associated with the log events for a [metric-schema] stanza. If the METRIC-SCHEMA-MEASURES setting does not extract a field as a measure field, the field appears in the metric data point as a dimension. Use the METRIC-SCHEMA-BLACKLIST-DIMS setting to specify fields in the log events that you do not want to appear as dimensions in the metric data points.

The syntax for this configuration looks like this:

[metric-schema:<unique_transforms_stanza_name>]
METRIC-SCHEMA-MEASURES = <your_measures_setting>
METRIC-SCHEMA-BLACKLIST-DIMS = <dimension_field1>, <dimension_field2>,...

If you have a field with both numeric and non-numeric values, the field is extracted as a measure by the _ALLNUMS_ setting and as a dimension due to the non-numeric values. If you want that field to be used only as a measure, blacklist it as a dimension field.

Configure log-to-metrics settings in props.conf

After configuring the metrics schema for a source type in transforms.conf, finish configuring the log-to-metrics settings in props.conf. Configure log-to-metrics settings in the props.conf file:

1. Reference the metric schema from transforms.conf.
2. Set up field extractions for the log data source.

Reference the metric schema from transforms.conf

To associate the log-to-metrics schema with a specific log source type, reference the transforms.conf configuration in the stanza for the log source type in props.conf. Use the METRIC-SCHEMA-TRANSFORMS setting, which has the following syntax:

[ <sourcetype> ]
METRIC-SCHEMA-TRANSFORMS = <metric-schema:stanza_name>[,<metric-schema:stanza_name>]...

Type the names of the log-to-metrics transform stanzas in the <stanza_name> part of the METRIC-SCHEMA-TRANSFORMS setting.

Set up field extractions for the log data source

To use log-to-metrics configurations, you must design a configuration that extracts fields from your log data. The configuration that you use depends on whether the data is structured or unstructured.

If your log data is in a structured format like a CSV file or JSON, add the INDEXED_EXTRACTIONS setting to the props.conf stanza. See Extract fields from files with structured data in Getting Data In.

If your log data is technically unstructured but is organized into field-value pairs that can easily be extracted, add TRANSFORMS-<class>=field_extraction to the stanza. This references the [field_extraction] stanza in transforms.conf, which is included by default with the Splunk platform. The [field_extraction] stanza uses a simple regular expression to extract field-value pairs from log data.

Order of operations for log-to-metrics conversion settings

The Splunk platform processes all METRIC-SCHEMA-MEASURES-<unique_metric_name_prefix> and METRIC-SCHEMA-BLACKLIST-DIMS-<unique_metric_name_prefix> settings ahead of basic METRIC-SCHEMA-MEASURES and METRIC-SCHEMA-BLACKLIST-DIMS settings.

In other words, the Splunk platform processes all of the event-targeting log-to-metrics settings before it processes the event-agnostic log-to-metrics settings. This allows the latter group of settings to process remaining events that were not targeted by the <unique_metric_name_prefix> settings.

Example of targeted log-to-metrics conversions

Use targeted log-to-metrics conversions when one log source type contains multiple event schemas with different sets of measurements and dimension fields. The following event collection example contains two event schemas. The events share a group field, and the values of group identify the two event schemas.

_time	Event
08-05-2017 20:26:29.073 -0700	INFO Metrics - group=queue, location=sf, corp=splunk, name=udp_queue, max_size_kb=0, current_size_kb=0, current_size=0, largest_size=0, smallest_size=0
08-05-2017 20:26:29.073 -0700	INFO Metrics - group=queue, location=sf, corp=splunk, name=aggqueue, max_size_kb=1024, current_size_kb=1, current_size=5, largest_size=35, smallest_size=0
08-05-2017 20:26:29.073 -0700	INFO Metrics - group=queue, location=sf, corp=splunk, name=auditqueue, max_size_kb=500, current_size_kb=0, current_size=0, largest_size=1, smallest_size=0
08-05-2017 20:26:29.075 -0700	INFO Metrics - group=pipeline, name=indexerpipe, processor=indexin, cpu_seconds=0, executes=171, cumulative_hits=2214401
08-05-2017 20:26:29.075 -0700	INFO Metrics - group=pipeline, name=indexerpipe, processor=index_thruput, cpu_seconds=0, executes=171, cumulative_hits=2214401
08-05-2017 20:26:29.075 -0700	INFO Metrics - group=pipeline, name=indexerpipe, processor=indexandforward, cpu_seconds=0, executes=171, cumulative_hits=2214401

After examining these events, you decide that you need to define a set of configurations in transforms.conf and props.conf that perform the following tasks:

• Set TRANSFORMS-<class>=field_extraction to extract field-value pairs from the log lines at ingest time.
• Use INGEST_EVAL to add a metric_name field to every event with a group field at ingest time. The new metric_name fields get the same values as their corresponding group fields.
• Provide separate log-to-metrics settings for the metric_name=queue events and the metric_name=pipeline events. Extract all of the numeric fields from the metric_name=queue events as measures.
• Blacklist the group, location, and corp fields from the dimensions for the metric_name=queue metric data points. Blacklist the group field from the dimensions for the metric_name=pipeline events.
• Associate the log-to-metrics settings with events that have the metrics_log source type.

Those configurations look as follows:

transforms.conf

[eval_pipeline]
INGEST_EVAL = metric_name=group

[metric-schema:extract_metrics]
METRIC-SCHEMA-MEASURES-queue=_ALLNUMS_
METRIC-SCHEMA-BLACKLIST-DIMS-queue=group,location,corp
METRIC-SCHEMA-MEASURES-pipeline=cpu_seconds,executes,cumulative_hits
METRIC-SCHEMA-BLACKLIST-DIMS-pipeline=group

props.conf

[metrics_log]
TRANSFORMS-fieldvalue=field_extraction
TRANSFORMS-metricslog=eval_pipeline
METRIC-SCHEMA-TRANSFORMS=metric-schema:extract_metrics

The metric data points created by these configurations look like the following examples:

_time	metric_name	_value	name	processor
08-05-2017 20:26:29.073 -0700	queue.max_size_kb	1024	aggqueue
08-05-2017 20:26:29.073 -0700	queue.current_size_kb	1	aggqueue
08-05-2017 20:26:29.073 -0700	queue.current_size	5	aggqueue
08-05-2017 20:26:29.073 -0700	queue.largest_size	35	aggqueue
08-05-2017 20:26:29.073 -0700	queue.smallest_size	0	aggqueue
08-05-2017 20:26:29.075 -0700	pipeline.cpu_seconds	0	indexerpipe	indexin
08-05-2017 20:26:29.075 -0700	pipeline.executes	171	indexerpipe	indexin
08-05-2017 20:26:29.075 -0700	pipeline.cumulative_hits	2214401	indexerpipe	indexin