Set up ingest-time log to metrics conversion with configuration files
If you have access to the props.conf
and transforms.conf
files for your deployment, you can manually configure log-to-metric transformations that are more sophisticated than the ones you can set up with Splunk Web. For example, you can design log-to-metrics transforms that can handle logs where not all of the events have the same sets of measurement and dimension fields.
To configure a logs-to-metrics conversion you need to add stanzas to your props.conf
and transforms.conf
files.
- Start by creating a
[metric-schema]
stanza with configurations for the measurements and blacklist dimensions in thetransforms.conf
file. - Then reference those configurations in a
props.conf
source type stanza.
For an overview of ingest-time conversion of logs to metric data points, see Convert event logs to metric data points.
Considerations for forwarders
When you process log-to-metric conversions, the type of forwarder that you are using and the type of data that you are ingesting govern the indexer version that you must use and the required location of the transforms.conf
and props.conf
files that have your log-to-metric configurations.
Structured data includes formats like CSV and JSON. For more information, see Additional configurations for extracting fields at index time.
Forwarder Version and Type | Type of Data | Indexer Version Required | Location of Log-to-Metrics Configuration Files |
---|---|---|---|
7.2x Universal Forwarder | Structured | 7.x | Universal Forwarder |
Any Universal Forwarder version | Unstructured | 7.2.x | Indexer |
7.2.x Heavy Forwarder | Structured | 7.x | Heavy Forwarder |
7.2.x Heavy Forwarder | Unstructured | 7.x | Heavy Forwarder |
The transforms.conf configurations
The basic transforms.conf
configurations give you the same log-to-metric capabilities as the Splunk Web method for defining log-to-metrics conversions. They enable you to create log-to-metric configurations for logs where all of the events have the same measurement and dimension fields.
In transforms.conf
, you need to set up a [metric-schema]
stanza that identifies lists of measurements and blacklist dimensions. The syntax for this configuration looks like this:
[metric-schema:<unique_transforms_stanza_name>] METRIC-SCHEMA-MEASURES = <measure_field1>, <measure_field2>,... METRIC-SCHEMA-BLACKLIST-DIMS = <dimension_field1>, <dimension_field2>,...
The METRIC-SCHEMA-MEASURES
and METRIC-SCHEMA-BLACKLIST-DIMS
settings determine how each of the log events associated with the stanza is transformed into multiple metric data points.
Setting syntax | Description | Required? |
---|---|---|
METRIC-SCHEMA-MEASURES = <measure_field1>, <measure_field2>,...
|
Provides a list of measurement fields. The Splunk platform generates a separate metric data point for each measurement field-value pair in an event associated with the [metric-schema] stanza. When this happens, the measurement field name becomes the metric_name value for the metric data point, and the measurement value becomes the _value value for the metric data point.
|
Yes |
METRIC-SCHEMA-BLACKLIST-DIMS = <dimension_field1>, <dimension_field2>,...
|
Provides a list of blacklist dimension fields. These are fields that should not appear as dimensions in the metric data points that are generated from an event associated with the [metric-schema] stanza. You might want to blacklist high-cardinality dimension fields that are unnecessary for your metric collection.
|
No |
All fields that are not identified as measure fields for METRIC-SCHEMA-MEASURES
or blacklist dimension fields for METRIC-SCHEMA-BLACKLIST-DIMS
appear in the metric data points as dimensions. A dimension field-value pair in an event is shared by all of the metric data fields generated from that event.
Apply log-to-metric settings to specific events in a log
Variants of the [metric-schema]
stanza settings enable you to create METRIC-SCHEMA-MEASURES
and METRIC-SCHEMA-BLACKLIST-DIMS
settings that are targeted to specific groups of log events according to the value of a field that is shared by all of the events in that log. Here is their syntax:
[metric-schema:<unique_transforms_stanza_name>] METRIC-SCHEMA-MEASURES-<unique_metric_name_prefix> = <measure_field1>, <measure_field2>,... METRIC-SCHEMA-BLACKLIST-DIMS-<unique_metric_name_prefix> = <dimension_field1>, <dimension_field2>,...
The <unique_metric_name_prefix>
must match the value of a metric_name
field that is shared by all of the events associated with the [metric-schema]
stanza. The values of the metric_name
field should correspond to the different event types present in the [metric-schema]
stanza.
If a metric_name
field is not already shared by your log events, there are ways to add it to your events. For example, you can:
- Create an index-time field extraction named
metric_name
. - Use the
INGEST_EVAL
setting to add ametric_name
field to the events at ingest time.
See Example of targeted log-to-metrics conversions. It shows you how to configure INGEST_EVAL
to add metric_name
to any event with a group
field.
When it is configured correctly, the METRIC-SCHEMA-MEASURES-<unique_metric_name_prefix>
setting produces metric data points with metric_name
values that follow this syntax: <unique_metric_name_prefix>.<measure_field_name>
.
Always use the METRIC-SCHEMA-BLACKLIST-DIMS-<unique_metric_name_prefix>
setting in conjunction with a corresponding METRIC-SCHEMA-MEASURES-<unique_metric_name_prefix>
setting.
The props.conf configuration
After you create the transforms.conf
configurations, you need to associate them with a source type in props.conf
. You do this by adding them to a stanza for the source type with a METRIC-SCHEMA-TRANSFORMS
setting. This setting has the following syntax:
[ <sourcetype> ] METRIC-SCHEMA-TRANSFORMS = <metric-schema:stanza_name>[,<metric-schema:stanza_name>]...
Place the names of your log-to-metrics transform stanzas in the <stanza_name>
part of the METRIC-SCHEMA-TRANSFORMS
configuration. This associates the log-to-metric transform stanza with the log events related to the source type.
Additional configurations for extracting fields at ingest time
The logs-to-metrics feature requires that you design a configuration that extracts fields from your log data. The configuration you use for this depends on whether the data is structured or unstructured.
If your log data is in a structured format like a CSV file or JSON, add the INDEXED_EXTRACTIONS
setting to the props.conf
stanza. For more information about the INDEXED_EXTRACTIONS
setting, see Extract fields from files with structured data in Getting Data In.
If your log data is technically unstructured, but its strings are organized into field-value pairs that can easily be extracted, add TRANSFORMS-<class>=field_extraction
to the stanza. This references the [field_extraction]
stanza in transforms.conf
, which is delivered by default with the Splunk platform. The [field_extraction]
stanza uses a simple regular expression to extract field-value pairs from log data.
Order of operations for log-to-metric conversion settings
The Splunk platform processes all METRIC-SCHEMA-MEASURES-<unique_metric_name_prefix>
and METRIC-SCHEMA-BLACKLIST-DIMS-<unique_metric_name_prefix>
settings ahead of basic METRIC-SCHEMA-MEASURES
and METRIC-SCHEMA-BLACKLIST-DIMS
settings.
In other words, the Splunk platform processes all of the event-targeting log-to-metrics settings before it processes the event-agnostic log-to-metrics settings. This allows the latter group of settings to process remaining events that were not targeted by the <unique_metric_name_prefix>
settings.
Example of targeted log-to-metrics conversions
Here is a collection of events. Notice that there are two event schemas with different sets of measurement and dimension fields. The events share a group
field, and the values of group
coordinate to the two event schemas.
_time | Event |
---|---|
08-05-2017 20:26:29.073 -0700 | INFO Metrics - group=queue, location=sf, corp=splunk, name=udp_queue, max_size_kb=0, current_size_kb=0, current_size=0, largest_size=0, smallest_size=0 |
08-05-2017 20:26:29.073 -0700 | INFO Metrics - group=queue, location=sf, corp=splunk, name=aggqueue, max_size_kb=1024, current_size_kb=1, current_size=5, largest_size=35, smallest_size=0 |
08-05-2017 20:26:29.073 -0700 | INFO Metrics - group=queue, location=sf, corp=splunk, name=auditqueue, max_size_kb=500, current_size_kb=0, current_size=0, largest_size=1, smallest_size=0 |
08-05-2017 20:26:29.075 -0700 | INFO Metrics - group=pipeline, name=indexerpipe, processor=indexin, cpu_seconds=0, executes=171, cumulative_hits=2214401 |
08-05-2017 20:26:29.075 -0700 | INFO Metrics - group=pipeline, name=indexerpipe, processor=index_thruput, cpu_seconds=0, executes=171, cumulative_hits=2214401 |
08-05-2017 20:26:29.075 -0700 | INFO Metrics - group=pipeline, name=indexerpipe, processor=indexandforward, cpu_seconds=0, executes=171, cumulative_hits=2214401 |
After examining these events, you decide you need to define a set of configurations in transforms.conf
and props.conf
that perform the following tasks:
- Set
TRANSFORMS-<class>=field_extraction
to extract field-value pairs from the log lines at ingest time. - Use
INGEST_EVAL
to add ametric_name
field to every event with agroup
field at ingest time. The newmetric_name
fields get the same values as their correspondinggroup
fields. - Provide separate log-to-metric settings for the
metric_name=queue
events and themetric_name=pipeline
events. - Blacklist the
group
,location
, andcorp
fields from themetric_name=queue
metric data points. Blacklist thegroup
field from themetric_name=pipeline
events. - Associate the log-to-metrics settings with events that have the
metrics_log
source type.
Those configurations would look like this:
transforms.conf
[eval_pipeline] INGEST_EVAL = metric_name=group [metric-schema:extract_metrics] METRIC-SCHEMA-MEASURES-queue=max_size_kb,current_size_kb,current_size,largest_size,smallest_size METRIC-SCHEMA-BLACKLIST-DIMS-queue=group,location,corp METRIC-SCHEMA-MEASURES-pipeline=cpu_seconds,executes,cumulative_hits METRIC-SCHEMA-BLACKLIST-DIMS-pipeline=group
props.conf
[metrics_log] TRANSFORMS-fieldvalue=field_extraction TRANSFORMS-metricslog=eval_pipeline METRIC-SCHEMA-TRANSFORMS=metric-schema:extract_metrics
Here are examples of the metric data points that these configurations would enable the Splunk platform to generate from those events:
_time | metric_name | _value | name | processor |
---|---|---|---|---|
08-05-2017 20:26:29.073 -0700 | queue.max_size_kb | 1024 | aggqueue | |
08-05-2017 20:26:29.073 -0700 | queue.current_size_kb | 1 | aggqueue | |
08-05-2017 20:26:29.073 -0700 | queue.current_size | 5 | aggqueue | |
08-05-2017 20:26:29.073 -0700 | queue.largest_size | 35 | aggqueue | |
08-05-2017 20:26:29.073 -0700 | queue.smallest_size | 0 | aggqueue | |
08-05-2017 20:26:29.075 -0700 | pipeline.cpu_seconds | 0 | indexerpipe | indexin |
08-05-2017 20:26:29.075 -0700 | pipeline.executes | 171 | indexerpipe | indexin |
08-05-2017 20:26:29.075 -0700 | pipeline.cumulative_hits | 2214401 | indexerpipe | indexin |
Set up ingest-time log-to-metrics conversion in Splunk Web | Search and monitor metrics |
This documentation applies to the following versions of Splunk® Enterprise: 7.2.0, 7.2.1, 7.2.2, 7.2.3, 7.2.4, 7.2.5, 7.2.6, 7.2.7, 7.2.8, 7.2.9, 7.2.10
Feedback submitted, thanks!