Set up ingest-time log to metrics conversion with configuration files
If you have access to the
transforms.conf files for your deployment, you can manually configure log-to-metric transformations that are more sophisticated than the ones you can set up with Splunk Web. For example, you can design log-to-metrics transforms that can handle logs where not all of the events have the same sets of measurement and dimension fields.
To configure a logs-to-metrics conversion you need to add stanzas to your
- Start by creating a
[metric-schema]stanza with configurations for the measurements and blacklist dimensions in the
- Then reference those configurations in a
props.confsource type stanza.
For an overview of ingest-time conversion of logs to metric data points, see Overview of log-to-metrics conversion.
Considerations for forwarders
When you process log-to-metric conversions, the type of forwarder that you are using and the type of data that you are ingesting govern the indexer version that you must use and the required location of the
props.conf files that have your log-to-metric configurations.
Structured data includes formats like CSV and JSON. For more information, see Additional configurations for extracting fields at index time.
|Forwarder Version and Type||Type of Data||Indexer Version Required||Location of Log-to-Metrics Configuration Files|
|7.2x Universal Forwarder||Structured||7.x||Universal Forwarder|
|Any Universal Forwarder version||Unstructured||7.2.x||Indexer|
|7.2.x Heavy Forwarder||Structured||7.x||Heavy Forwarder|
|7.2.x Heavy Forwarder||Unstructured||7.x||Heavy Forwarder|
The transforms.conf configurations
transforms.conf configurations give you the same log-to-metric capabilities as the Splunk Web method for defining log-to-metrics conversions. They enable you to create log-to-metric configurations for logs where all of the events have the same measurement and dimension fields.
transforms.conf, you need to set up a
[metric-schema] stanza that identifies lists of measurements and blacklist dimensions. The syntax for this configuration looks like this:
[metric-schema:<unique_transforms_stanza_name>] METRIC-SCHEMA-MEASURES = <measure_field1>, <measure_field2>,... METRIC-SCHEMA-BLACKLIST-DIMS = <dimension_field1>, <dimension_field2>,...
METRIC-SCHEMA-BLACKLIST-DIMS settings determine how each of the log events associated with the stanza is transformed into multiple metric data points.
|| Provides a list of measurement fields. The Splunk platform generates a separate metric data point for each measurement field-value pair in an event associated with the
|| Provides a list of blacklist dimension fields. These are fields that should not appear as dimensions in the metric data points that are generated from an event associated with the
All fields that are not identified as measure fields for
METRIC-SCHEMA-MEASURES or blacklist dimension fields for
METRIC-SCHEMA-BLACKLIST-DIMS appear in the metric data points as dimensions. A dimension field-value pair in an event is shared by all of the metric data fields generated from that event.
Apply log-to-metric settings to specific events in a log
Variants of the
[metric-schema] stanza settings enable you to create
METRIC-SCHEMA-BLACKLIST-DIMS settings that are targeted to specific groups of log events according to the value of a field that is shared by all of the events in that log. Here is their syntax:
[metric-schema:<unique_transforms_stanza_name>] METRIC-SCHEMA-MEASURES-<unique_metric_name_prefix> = <measure_field1>, <measure_field2>,... METRIC-SCHEMA-BLACKLIST-DIMS-<unique_metric_name_prefix> = <dimension_field1>, <dimension_field2>,...
<unique_metric_name_prefix> must match the value of a
metric_name field that is shared by all of the events associated with the
[metric-schema] stanza. The values of the
metric_name field should correspond to the different event types present in the
metric_name field is not already shared by your log events, there are ways to add it to your events. For example, you can:
- Create an index-time field extraction named
- Use the
INGEST_EVALsetting to add a
metric_namefield to the events at ingest time.
See Example of targeted log-to-metrics conversions. It shows you how to configure
INGEST_EVAL to add
metric_name to any event with a
When it is configured correctly, the
METRIC-SCHEMA-MEASURES-<unique_metric_name_prefix> setting produces metric data points with
metric_name values that follow this syntax:
Always use the
METRIC-SCHEMA-BLACKLIST-DIMS-<unique_metric_name_prefix> setting in conjunction with a corresponding
The props.conf configuration
After you create the
transforms.conf configurations, you need to associate them with a source type in
props.conf. You do this by adding them to a stanza for the source type with a
METRIC-SCHEMA-TRANSFORMS setting. This setting has the following syntax:
[ <sourcetype> ] METRIC-SCHEMA-TRANSFORMS = <metric-schema:stanza_name>[,<metric-schema:stanza_name>]...
Place the names of your log-to-metrics transform stanzas in the
<stanza_name> part of the
METRIC-SCHEMA-TRANSFORMS configuration. This associates the log-to-metric transform stanza with the log events related to the source type.
Additional configurations for extracting fields at ingest time
The logs-to-metrics feature requires that you design a configuration that extracts fields from your log data. The configuration you use for this depends on whether the data is structured or unstructured.
If your log data is in a structured format like a CSV file or JSON, add the
INDEXED_EXTRACTIONS setting to the
props.conf stanza. For more information about the
INDEXED_EXTRACTIONS setting, see Extract fields from files with structured data in Getting Data In.
If your log data is technically unstructured, but its strings are organized into field-value pairs that can easily be extracted, add
TRANSFORMS-<class>=field_extraction to the stanza. This references the
[field_extraction] stanza in
transforms.conf, which is delivered by default with the Splunk platform. The
[field_extraction] stanza uses a simple regular expression to extract field-value pairs from log data.
Order of operations for log-to-metric conversion settings
The Splunk platform processes all
METRIC-SCHEMA-BLACKLIST-DIMS-<unique_metric_name_prefix> settings ahead of basic
In other words, the Splunk platform processes all of the event-targeting log-to-metrics settings before it processes the event-agnostic log-to-metrics settings. This allows the latter group of settings to process remaining events that were not targeted by the
Example of targeted log-to-metrics conversions
Here is a collection of events. Notice that there are two event schemas with different sets of measurement and dimension fields. The events share a
group field, and the values of
group coordinate to the two event schemas.
|08-05-2017 20:26:29.073 -0700||INFO Metrics - group=queue, location=sf, corp=splunk, name=udp_queue, max_size_kb=0, current_size_kb=0, current_size=0, largest_size=0, smallest_size=0|
|08-05-2017 20:26:29.073 -0700||INFO Metrics - group=queue, location=sf, corp=splunk, name=aggqueue, max_size_kb=1024, current_size_kb=1, current_size=5, largest_size=35, smallest_size=0|
|08-05-2017 20:26:29.073 -0700||INFO Metrics - group=queue, location=sf, corp=splunk, name=auditqueue, max_size_kb=500, current_size_kb=0, current_size=0, largest_size=1, smallest_size=0|
|08-05-2017 20:26:29.075 -0700||INFO Metrics - group=pipeline, name=indexerpipe, processor=indexin, cpu_seconds=0, executes=171, cumulative_hits=2214401|
|08-05-2017 20:26:29.075 -0700||INFO Metrics - group=pipeline, name=indexerpipe, processor=index_thruput, cpu_seconds=0, executes=171, cumulative_hits=2214401|
|08-05-2017 20:26:29.075 -0700||INFO Metrics - group=pipeline, name=indexerpipe, processor=indexandforward, cpu_seconds=0, executes=171, cumulative_hits=2214401|
After examining these events, you decide you need to define a set of configurations in
props.conf that perform the following tasks:
TRANSFORMS-<class>=field_extractionto extract field-value pairs from the log lines at ingest time.
INGEST_EVALto add a
metric_namefield to every event with a
groupfield at ingest time. The new
metric_namefields get the same values as their corresponding
- Provide separate log-to-metric settings for the
metric_name=queueevents and the
- Blacklist the
corpfields from the
metric_name=queuemetric data points. Blacklist the
groupfield from the
- Associate the log-to-metrics settings with events that have the
Those configurations would look like this:
[eval_pipeline] INGEST_EVAL = metric_name=group [metric-schema:extract_metrics] METRIC-SCHEMA-MEASURES-queue=max_size_kb,current_size_kb,current_size,largest_size,smallest_size METRIC-SCHEMA-BLACKLIST-DIMS-queue=group,location,corp METRIC-SCHEMA-MEASURES-pipeline=cpu_seconds,executes,cumulative_hits METRIC-SCHEMA-BLACKLIST-DIMS-pipeline=group
[metrics_log] TRANSFORMS-fieldvalue=field_extraction TRANSFORMS-metricslog=eval_pipeline METRIC-SCHEMA-TRANSFORMS=metric-schema:extract_metrics
Here are examples of the metric data points that these configurations would enable the Splunk platform to generate from those events:
|08-05-2017 20:26:29.073 -0700||queue.max_size_kb||1024||aggqueue|
|08-05-2017 20:26:29.073 -0700||queue.current_size_kb||1||aggqueue|
|08-05-2017 20:26:29.073 -0700||queue.current_size||5||aggqueue|
|08-05-2017 20:26:29.073 -0700||queue.largest_size||35||aggqueue|
|08-05-2017 20:26:29.073 -0700||queue.smallest_size||0||aggqueue|
|08-05-2017 20:26:29.075 -0700||pipeline.cpu_seconds||0||indexerpipe||indexin|
|08-05-2017 20:26:29.075 -0700||pipeline.executes||171||indexerpipe||indexin|
|08-05-2017 20:26:29.075 -0700||pipeline.cumulative_hits||2214401||indexerpipe||indexin|
Set up ingest-time log-to-metrics conversion in Splunk Web
Search and monitor metrics
This documentation applies to the following versions of Splunk® Enterprise: 7.2.0, 7.2.1, 7.2.2, 7.2.3, 7.2.4, 7.2.5, 7.2.6, 7.2.7