Set up ingest-time log to metrics conversion with configuration files

If you have access to the props.conf and transforms.conf files for your deployment, you can manually configure log-to-metric transformations that are more sophisticated than the ones you can set up with Splunk Web. For example, you can design log-to-metrics transforms that can handle logs where not all of the events have the same sets of measurement and dimension fields.

To configure a logs-to-metrics conversion you need to add stanzas to your props.conf and transforms.conf files.

1. Start by creating a [metric-schema] stanza with configurations for the measurements and blacklist dimensions in the transforms.conf file.
2. Then reference those configurations in a props.conf source type stanza.

For an overview of ingest-time conversion of logs to metric data points, see Convert event logs to metric data points.

Considerations for forwarders

When you process log-to-metric conversions, the type of forwarder that you are using and the type of data that you are ingesting govern the indexer version that you must use and the required location of the transforms.conf and props.conf files that have your log-to-metric configurations.

Structured data includes formats like CSV and JSON. For more information, see Additional configurations for extracting fields at index time.

Forwarder Version and Type	Type of Data	Indexer Version Required	Location of Log-to-Metrics Configuration Files
7.2x Universal Forwarder	Structured	7.x	Universal Forwarder
Any Universal Forwarder version	Unstructured	7.2.x	Indexer
7.2.x Heavy Forwarder	Structured	7.x	Heavy Forwarder
7.2.x Heavy Forwarder	Unstructured	7.x	Heavy Forwarder

The transforms.conf configurations

The basic transforms.conf configurations give you the same log-to-metric capabilities as the Splunk Web method for defining log-to-metrics conversions. They enable you to create log-to-metric configurations for logs where all of the events have the same measurement and dimension fields.

In transforms.conf, you need to set up a [metric-schema] stanza that identifies lists of measurements and blacklist dimensions. The syntax for this configuration looks like this:

[metric-schema:<unique_transforms_stanza_name>]
METRIC-SCHEMA-MEASURES = <measure_field1>, <measure_field2>,...
METRIC-SCHEMA-BLACKLIST-DIMS = <dimension_field1>, <dimension_field2>,...

The METRIC-SCHEMA-MEASURES and METRIC-SCHEMA-BLACKLIST-DIMS settings determine how each of the log events associated with the stanza is transformed into multiple metric data points.

Setting syntax	Description	Required?
METRIC-SCHEMA-MEASURES = <measure_field1>, <measure_field2>,...	Provides a list of measurement fields. The Splunk platform generates a separate metric data point for each measurement field-value pair in an event associated with the [metric-schema] stanza. When this happens, the measurement field name becomes the metric_name value for the metric data point, and the measurement value becomes the _value value for the metric data point.	Yes
METRIC-SCHEMA-BLACKLIST-DIMS = <dimension_field1>, <dimension_field2>,...	Provides a list of blacklist dimension fields. These are fields that should not appear as dimensions in the metric data points that are generated from an event associated with the [metric-schema] stanza. You might want to blacklist high-cardinality dimension fields that are unnecessary for your metric collection.	No

All fields that are not identified as measure fields for METRIC-SCHEMA-MEASURES or blacklist dimension fields for METRIC-SCHEMA-BLACKLIST-DIMS appear in the metric data points as dimensions. A dimension field-value pair in an event is shared by all of the metric data fields generated from that event.

Apply log-to-metric settings to specific events in a log

Variants of the [metric-schema] stanza settings enable you to create METRIC-SCHEMA-MEASURES and METRIC-SCHEMA-BLACKLIST-DIMS settings that are targeted to specific groups of log events according to the value of a field that is shared by all of the events in that log. Here is their syntax:

[metric-schema:<unique_transforms_stanza_name>]
METRIC-SCHEMA-MEASURES-<unique_metric_name_prefix> = <measure_field1>, <measure_field2>,...
METRIC-SCHEMA-BLACKLIST-DIMS-<unique_metric_name_prefix> = <dimension_field1>, <dimension_field2>,...

The <unique_metric_name_prefix> must match the value of a metric_name field that is shared by all of the events associated with the [metric-schema] stanza. The values of the metric_name field should correspond to the different event types present in the [metric-schema] stanza.

If a metric_name field is not already shared by your log events, there are ways to add it to your events. For example, you can:

• Create an index-time field extraction named metric_name.
• Use the INGEST_EVAL setting to add a metric_name field to the events at ingest time.

See Example of targeted log-to-metrics conversions. It shows you how to configure INGEST_EVAL to add metric_name to any event with a group field.

When it is configured correctly, the METRIC-SCHEMA-MEASURES-<unique_metric_name_prefix> setting produces metric data points with metric_name values that follow this syntax: <unique_metric_name_prefix>.<measure_field_name>.

Always use the METRIC-SCHEMA-BLACKLIST-DIMS-<unique_metric_name_prefix> setting in conjunction with a corresponding METRIC-SCHEMA-MEASURES-<unique_metric_name_prefix> setting.

The props.conf configuration

After you create the transforms.conf configurations, you need to associate them with a source type in props.conf. You do this by adding them to a stanza for the source type with a METRIC-SCHEMA-TRANSFORMS setting. This setting has the following syntax:

[ <sourcetype> ]
METRIC-SCHEMA-TRANSFORMS = <metric-schema:stanza_name>[,<metric-schema:stanza_name>]...

Place the names of your log-to-metrics transform stanzas in the <stanza_name> part of the METRIC-SCHEMA-TRANSFORMS configuration. This associates the log-to-metric transform stanza with the log events related to the source type.

Additional configurations for extracting fields at ingest time

The logs-to-metrics feature requires that you design a configuration that extracts fields from your log data. The configuration you use for this depends on whether the data is structured or unstructured.

If your log data is in a structured format like a CSV file or JSON, add the INDEXED_EXTRACTIONS setting to the props.conf stanza. For more information about the INDEXED_EXTRACTIONS setting, see Extract fields from files with structured data in Getting Data In.

If your log data is technically unstructured, but its strings are organized into field-value pairs that can easily be extracted, add TRANSFORMS-<class>=field_extraction to the stanza. This references the [field_extraction] stanza in transforms.conf, which is delivered by default with the Splunk platform. The [field_extraction] stanza uses a simple regular expression to extract field-value pairs from log data.

Order of operations for log-to-metric conversion settings

The Splunk platform processes all METRIC-SCHEMA-MEASURES-<unique_metric_name_prefix> and METRIC-SCHEMA-BLACKLIST-DIMS-<unique_metric_name_prefix> settings ahead of basic METRIC-SCHEMA-MEASURES and METRIC-SCHEMA-BLACKLIST-DIMS settings.

In other words, the Splunk platform processes all of the event-targeting log-to-metrics settings before it processes the event-agnostic log-to-metrics settings. This allows the latter group of settings to process remaining events that were not targeted by the <unique_metric_name_prefix> settings.

Example of targeted log-to-metrics conversions

Here is a collection of events. Notice that there are two event schemas with different sets of measurement and dimension fields. The events share a group field, and the values of group coordinate to the two event schemas.

_time	Event
08-05-2017 20:26:29.073 -0700	INFO Metrics - group=queue, location=sf, corp=splunk, name=udp_queue, max_size_kb=0, current_size_kb=0, current_size=0, largest_size=0, smallest_size=0
08-05-2017 20:26:29.073 -0700	INFO Metrics - group=queue, location=sf, corp=splunk, name=aggqueue, max_size_kb=1024, current_size_kb=1, current_size=5, largest_size=35, smallest_size=0
08-05-2017 20:26:29.073 -0700	INFO Metrics - group=queue, location=sf, corp=splunk, name=auditqueue, max_size_kb=500, current_size_kb=0, current_size=0, largest_size=1, smallest_size=0
08-05-2017 20:26:29.075 -0700	INFO Metrics - group=pipeline, name=indexerpipe, processor=indexin, cpu_seconds=0, executes=171, cumulative_hits=2214401
08-05-2017 20:26:29.075 -0700	INFO Metrics - group=pipeline, name=indexerpipe, processor=index_thruput, cpu_seconds=0, executes=171, cumulative_hits=2214401
08-05-2017 20:26:29.075 -0700	INFO Metrics - group=pipeline, name=indexerpipe, processor=indexandforward, cpu_seconds=0, executes=171, cumulative_hits=2214401

After examining these events, you decide you need to define a set of configurations in transforms.conf and props.conf that perform the following tasks:

• Set TRANSFORMS-<class>=field_extraction to extract field-value pairs from the log lines at ingest time.
• Use INGEST_EVAL to add a metric_name field to every event with a group field at ingest time. The new metric_name fields get the same values as their corresponding group fields.
• Provide separate log-to-metric settings for the metric_name=queue events and the metric_name=pipeline events.
• Blacklist the group, location, and corp fields from the metric_name=queue metric data points. Blacklist the group field from the metric_name=pipeline events.
• Associate the log-to-metrics settings with events that have the metrics_log source type.

Those configurations would look like this:

transforms.conf

[eval_pipeline]
INGEST_EVAL = metric_name=group

[metric-schema:extract_metrics]
METRIC-SCHEMA-MEASURES-queue=max_size_kb,current_size_kb,current_size,largest_size,smallest_size
METRIC-SCHEMA-BLACKLIST-DIMS-queue=group,location,corp
METRIC-SCHEMA-MEASURES-pipeline=cpu_seconds,executes,cumulative_hits
METRIC-SCHEMA-BLACKLIST-DIMS-pipeline=group

props.conf

[metrics_log]
TRANSFORMS-fieldvalue=field_extraction
TRANSFORMS-metricslog=eval_pipeline
METRIC-SCHEMA-TRANSFORMS=metric-schema:extract_metrics

Here are examples of the metric data points that these configurations would enable the Splunk platform to generate from those events:

_time	metric_name	_value	name	processor
08-05-2017 20:26:29.073 -0700	queue.max_size_kb	1024	aggqueue
08-05-2017 20:26:29.073 -0700	queue.current_size_kb	1	aggqueue
08-05-2017 20:26:29.073 -0700	queue.current_size	5	aggqueue
08-05-2017 20:26:29.073 -0700	queue.largest_size	35	aggqueue
08-05-2017 20:26:29.073 -0700	queue.smallest_size	0	aggqueue
08-05-2017 20:26:29.075 -0700	pipeline.cpu_seconds	0	indexerpipe	indexin
08-05-2017 20:26:29.075 -0700	pipeline.executes	171	indexerpipe	indexin
08-05-2017 20:26:29.075 -0700	pipeline.cumulative_hits	2214401	indexerpipe	indexin