Set up ingest-time log-to-metrics conversion with configuration files
If you have access to the props.conf
and transforms.conf
files for your deployment, you can manually configure log-to-metrics transformations that are more sophisticated than the ones you can set up with Splunk Web. For example, you can design log-to-metrics transformations that can handle logs where not all of the events have the same sets of measurement and dimension fields.
To configure a log-to-metrics conversion, you need to add stanzas to your props.conf
and transforms.conf
files.
- Specify the schema for log-to-metrics transformations in
transforms.conf
. - Configure log-to-metrics settings in
props.conf
.
For an overview of ingest-time conversion of logs to metric data points, see Convert event logs to metric data points.
Log-to-metric feature extensions that are not available in Splunk Web
When you manage log-to-metric processing through direct edits to configuration files, you can take advantage of optional feature extensions that are not yet available in Splunk Web.
Extended feature | In Splunk Web | Through configuration file edits | Setting |
---|---|---|---|
Sort events by the values of a shared field and target different log-to-metric conversion rules to each set of events. | The log-to-metric conversion rules are applied to all events that belong to the selected source type. | If all of the events belonging to the selected source type all share a metric_name field, you can devise log-to-metric rules for events with specific values of that field.
|
METRIC-SCHEMA-MEASURES-<unique_metric_name_prefix> , METRIC-SCHEMA-BLACKLIST-DIMS-<unique_metric_name_prefix> , and METRIC-SCHEMA-WHITELIST-DIMS-<unique_metric_name_prefix> .
|
Considerations for forwarders
When processing log-to-metrics conversions, the type of forwarder that you are using and the type of data that you are ingesting require specific indexer versions and locations for the transforms.conf
and props.conf
files with the log-to-metrics configurations.
The following configurations apply whether the data involved is structured or unstructured. Structured data includes formats like CSV and JSON. For more information, see Set up field extractions for the log data source.
Type of data | Forwarder version and type | Indexer version required | Location of log-to-metrics configuration files |
---|---|---|---|
Structured | Universal Forwarder with version 7.3.0 or later (but earlier than 8.x) | 7.x or above | Universal Forwarder |
Unstructured | Any Universal Forwarder version earlier than 8.x | 7.3.x or above | Indexer |
Structured | 8.x Universal Forwarder | 8.x | Universal Forwarder |
Unstructured | 8.x Universal Forwarder | 8.x | Indexer |
Structured | 7.3.x Heavy Forwarder | 7.x or above | Heavy Forwarder |
Unstructured | 7.3.x Heavy Forwarder | 7.x or above | Heavy Forwarder |
Structured | 8.x Heavy Forwarder | 8.x | Heavy Forwarder |
Unstructured | 8.x Heavy Forwarder | 8.x | Heavy Forwarder |
Specify the schema for log-to-metrics transformations in transforms.conf
Use configurations in the transforms.conf
file to identify which events in a log contain metrics data points that you want to extract, and then specify how to extract the metrics from the log events.
- Identify which events in a log contain metrics data points that you want to extract, and then apply the relevant settings in the configuration.
- Specify how to extract measures from log events.
- Determine which event fields are transformed into metric dimensions.
Log-to-metrics metric schema settings reference
The metric schema settings determine how the log events associated with the stanza are transformed into metric data points. This table describes the syntax for the available settings when configuring log-to-metrics in transforms.conf
:
Metric schema setting syntax | Description | Required? |
---|---|---|
METRIC-SCHEMA-MEASURES = (_ALLNUMS_ | (_NUMS_EXCEPT_ )? <field1>, <field2>,... )
|
Identifies how to extract numeric fields in events as measures in the metric data points that correspond to those events. You can identify the numeric fields that should be turned into measures, or you can process all numeric fields in an event as measures. | Yes |
METRIC-SCHEMA-BLACKLIST-DIMS = <dimension_field1>, <dimension_field2>,...
|
Identifies excluded dimension fields. These are fields in your event data that cannot appear as dimensions in the metric data points that are generated from an event associated with the [metric-schema] stanza. You might want to set up a blacklist if some of the fields in your event data are high-cardinality dimension fields that are unnecessary for your metric collection.
|
No |
METRIC-SCHEMA-WHITELIST-DIMS = <dimension_field1>,<dimension_field2>,...
|
Identifies whitelisted dimension fields. These are fields in your event data that must appear as dimensions in the metric data points that are generated from an event associated with the [metric-schema] stanza. You might want to set up a whitelist if most of the fields in your event data are high-cardinality or otherwise unnecessary for your metrics.
|
No |
Apply log-to-metrics settings to all events in a log
Use the METRIC-SCHEMA-MEASURES
setting to apply log-to-metrics processing to all events in a log. You can optionally use the METRIC-SCHEMA-BLACKLIST-DIMS
and METRIC-SCHEMA-WHITELIST-DIMS
settings to filter unnecessary dimension fields out of the resulting metric data points.
This configuration syntax is as follows:
[metric-schema:<unique_transforms_stanza_name] METRIC-SCHEMA-MEASURES = (_ALLNUMS_ | (_NUMS_EXCEPT_ )? <field1>, <field2>,... ) METRIC-SCHEMA-BLACKLIST-DIMS = <dimension_field1>, <dimension_field2>,... METRIC-SCHEMA-WHITELIST-DIMS = <dimension_field1>,<dimension_field2>,...
Replace (_ALLNUMS_ | (_NUMS_EXCEPT_ )? <field1>, <field2>,... )
with the specific setting you choose from Specify how to extract metrics from log events.
Apply log-to-metrics settings to specific events in a log
Use the METRIC-SCHEMA-MEASURES-<unique_metric_name_prefix>
setting to apply log-to-metrics processing to specific events within a log. It will target a subset of events according to the value of a field shared by all of the events in the log.
Optionally use the METRIC-SCHEMA-BLACKLIST-DIMS-<unique_metric_name_prefix>
and METRIC-SCHEMA-WHITELIST-DIMS-<unique_metric_name_prefix>
parameters to filter unnecessary dimension fields out of the resulting metric data points.
This configuration syntax is as follows:
[metric-schema:<unique_transforms_stanza_name>] METRIC-SCHEMA-MEASURES-<unique_metric_name_prefix> = (_ALLNUMS_ | (_NUMS_EXCEPT_ )? <field1>, <field2>,... ) METRIC-SCHEMA-BLACKLIST-DIMS-<unique_metric_name_prefix> = <dimension_field1>, <dimension_field2>,... METRIC-SCHEMA-WHITELIST-DIMS-<unique_metric_name_prefix> = <dimension_field1>,<dimension_field2>,...
Replace (_ALLNUMS_ | (_NUMS_EXCEPT_ )? <field1>, <field2>,... )
with the specific setting you choose from Specify how to extract metrics from log events.
The <unique_metric_name_prefix>
must match the value of a metric_name
field that is shared by all of the events associated with the [metric-schema]
stanza. The values of the metric_name
field must correspond to the different event types present in the metric-schema
stanza.
If a metric_name
field is not already shared by your log events, there are ways to add it to your events:
- Create an index-time field extraction named
metric_name
. - Use the
INGEST_EVAL
setting to add ametric_name
field to the events at ingest time. For an example that shows you how to configure this, see Example of targeted log-to-metrics conversions.
When configured correctly, the METRIC-SCHEMA-MEASURES-<unique_metric_name_prefix>
setting produces metric data points with measurements that follow this syntax: metric_name:<unique_metric_name_prefix>.<measure_field_name>=<numeric_measure_field_value>
.
Always use the METRIC-SCHEMA-BLACKLIST-DIMS-<unique_metric_name_prefix>
and METRIC-SCHEMA-WHITELIST-DIMS-<unique_metric_name_prefix>
settings in conjunction with a corresponding METRIC-SCHEMA-MEASURES-<unique_metric_name_prefix>
setting.
Specify how to extract measures from log events
There are several options to extract measures from log events:
- You can extract all numeric fields in events as measures.
- You can extract numeric fields with some exclusions as measures, or exclude specific fields from being extracted as measures.
- You can extract specific fields as measures, or whitelist specific fields to be extracted as measures.
These options are available whether you apply log-to-metrics settings to all events in a log or only to specific events in a log.
Method for extracting measures | Description | Syntax example | Fields with numeric and non-numeric values |
---|---|---|---|
Extract all numeric fields as measures. | Set up a [metric-schema] stanza using the _ALLNUMS_ setting.
|
[metric-schema:<unique_transforms_stanza_name] METRIC-SCHEMA-MEASURES = _ALLNUMS_ |
The _ALLNUMS_ setting extracts numeric values as measures for the field. Due to the non-numeric values, the same field is also used as a dimension field. If you want that field to be used only as a measure, exclude it as a dimension field. See Determine which event fields are transformed into metric dimensions.
|
Extract numeric fields with some exclusions as measures. | Set up a [metric-schema] stanza using the _NUMS_EXCEPT_ setting to define a blacklist of fields that you do not want extracted as measures. You must have a space between _NUMS_EXCEPT_ and the field name for the setting to function.
|
[metric-schema:<unique_transforms_stanza_name] METRIC-SCHEMA-MEASURES = _NUMS_EXCEPT_ <measure_field1>, <measure_field2>,... |
The _NUMS_EXCEPT_ setting extracts the numeric values as measures for the field. If you want a field with both numeric and non-numeric fields to only be a dimension field, exclude it from being extracted as a measure using the _NUMS_EXCEPT_ setting.
|
Extract specific fields as measures. | In transforms.conf , set up a [metric-schema] stanza that identifies lists of fields that contain measurement values to extract only those fields as measures.
|
[metric-schema:<unique_transforms_stanza_name>] METRIC-SCHEMA-MEASURES = <measure_field1>, <measure_field2>,... |
If you specify a field that has both numeric and non-numeric values with this setting, the numeric values are extracted as measures and the non-numeric values are ignored. The field is not used as a dimension field with the non-numeric values. |
You can use the wildcard character to match multiple similar numeric fields in your data. For example, say your event data contains the following numeric fields max_size_kb
, min_size_kb
, and current_size_kb
. You can set a <measure_field>
value of *_size_kb
to include all three of those numeric fields in the list of measures without listing each one separately.
Determine which event fields are transformed into metric dimensions
Any event field that the METRIC-SCHEMA-MEASURES
setting does not identify as a measure can appear in the metric data point that is generated from that event as a dimension.
However, you can optionally use the METRIC-SCHEMA-BLACKLIST-DIMS
and METRIC-SCHEMA-WHITELIST-DIMS
settings to filter dimensions out of the metric data point.
- If you provide a list of event fields for
METRIC-SCHEMA-BLACKLIST-DIMS
, the search head transforms all unlisted non-measure fields into metric data point dimensions. - If you provide a list of event fields for
METRIC-SCHEMA-WHITELIST-DIMS
, the fields in that list are the only fields that the search head transforms into metric data point dimensions. It ignores all other non-measure fields
The syntax for this configuration looks like this:
[metric-schema:<unique_transforms_stanza_name>] METRIC-SCHEMA-MEASURES = <your_measures_setting> METRIC-SCHEMA-BLACKLIST-DIMS = <dimension_field1>, <dimension_field2>,... METRIC-SCHEMA-WHITELIST-DIMS = <dimension_field1>, <dimension_field2>,...
The search processor uses the following evaluation logic when a metric schema has fields defined for both METRIC-SCHEMA-BLACKLIST-DIMS
and METRIC-SCHEMA-WHITELIST-DIMS
:
- If a dimension is listed in the
BLACKLIST
, it won't be present in the resulting metric data point, even if it also appears in theWHITELIST
. - If a dimension is not listed in the
WHITELIST
, it won't be present in the resulting metric data point, even if it also does not appear in theBLACKLIST
.
You can use the wildcard character to match multiple similar dimension fields in your data. For example, say your event data contains the following dimensions customer_id
, employee_id
, and consultant_id
. You can set a <dimension_name>
value of *_id
to include all three of those dimensions in the dimension field list without listing each one separately.
If you have a field with both numeric and non-numeric values and you are using the _ALNUMS_
setting or the _NUMS_EXCEPT_
setting, the field is extracted as a measure by the setting and again as a dimension due to the non-numeric values. If you want that field to be used only as a measure, exclude it as a dimension field.
About extracting new fields from unstructured data or renaming fields
The Log to Metrics feature works by going over indexed fields in _meta
to convert them to metric measurements or include or exclude them as dimensions. If you use structured data indexed field extraction methods (such as CSV or JSON extractions) you do not have to do extra setup beyond what we have provided here. However, if you are designing a more complex configuration that involves the extraction of fields from unstructured data, or the renaming of existing fields, you will need to make sure the fields are indexed. You can do this by setting WRITE_META=true
in the transforms.conf stanza for the extraction.
See How Splunk software builds indexed fields, in Getting Data In.
Configure log-to-metrics settings in props.conf
After configuring the metrics schema for a source type in transforms.conf
, finish configuring the log-to-metrics settings in props.conf
.
Configure log-to-metrics settings in the props.conf
file:
- Reference the metric schema from
transforms.conf
. - Set up field extractions for the log data source.
Reference the metric schema from transforms.conf
To associate the log-to-metrics schema with a specific log source type, reference the transforms.conf
configuration in the stanza for the log source type in props.conf
. Use the METRIC-SCHEMA-TRANSFORMS
setting, which has the following syntax:
[ <sourcetype> ] METRIC-SCHEMA-TRANSFORMS = <metric-schema:stanza_name>[,<metric-schema:stanza_name>]...
Type the names of the log-to-metrics transform stanzas in the <stanza_name>
part of the METRIC-SCHEMA-TRANSFORMS
setting.
Set up field extractions for the log data source
To use log-to-metrics configurations, you must design a configuration that extracts fields from your log data. The configuration that you use depends on whether the data is structured or unstructured.
If your log data is in a structured format like a CSV file or JSON, add the INDEXED_EXTRACTIONS
setting to the props.conf
stanza. See Extract fields from files with structured data in Getting Data In.
If your log data is technically unstructured but is organized into field-value pairs that can easily be extracted, add TRANSFORMS-<class>=field_extraction
to the stanza. This references the [field_extraction]
stanza in transforms.conf
, which is included by default with the Splunk platform. The [field_extraction]
stanza uses a simple regular expression to extract field-value pairs from log data.
Order of operations for log-to-metrics conversion settings
The Splunk platform processes all METRIC-SCHEMA-MEASURES-<unique_metric_name_prefix>
and METRIC-SCHEMA-BLACKLIST-DIMS-<unique_metric_name_prefix>
settings ahead of basic METRIC-SCHEMA-MEASURES
and METRIC-SCHEMA-BLACKLIST-DIMS
settings.
In other words, the Splunk platform processes all of the event-targeting log-to-metrics settings before it processes the event-agnostic log-to-metrics settings. This allows the latter group of settings to process remaining events that were not targeted by the <unique_metric_name_prefix>
settings.
Example of targeted log-to-metrics conversions
Use targeted log-to-metrics conversions when one log source type contains multiple event schemas with different sets of measurements and dimension fields. The following event collection example contains two event schemas. The events share a group
field, and the values of group
identify the two event schemas.
_time | Event |
---|---|
08-05-2017 20:26:29.073 -0700 | INFO Metrics - group=queue, location=sf, corp=splunk, name=udp_queue, max_size_kb=0, current_size_kb=0, current_size=0, largest_size=0, smallest_size=0 |
08-05-2017 20:26:29.073 -0700 | INFO Metrics - group=queue, location=sf, corp=splunk, name=aggqueue, max_size_kb=1024, current_size_kb=1, current_size=5, largest_size=35, smallest_size=0 |
08-05-2017 20:26:29.073 -0700 | INFO Metrics - group=queue, location=sf, corp=splunk, name=auditqueue, max_size_kb=500, current_size_kb=0, current_size=0, largest_size=1, smallest_size=0 |
08-05-2017 20:26:29.075 -0700 | INFO Metrics - group=pipeline, name=indexerpipe, processor=indexin, cpu_seconds=0, executes=171, cumulative_hits=2214401 |
08-05-2017 20:26:29.075 -0700 | INFO Metrics - group=pipeline, name=indexerpipe, processor=index_thruput, cpu_seconds=0, executes=171, cumulative_hits=2214401 |
08-05-2017 20:26:29.075 -0700 | INFO Metrics - group=pipeline, name=indexerpipe, processor=indexandforward, cpu_seconds=0, executes=171, cumulative_hits=2214401 |
After examining these events, you decide that you need to define a set of configurations in transforms.conf
and props.conf
that perform the following tasks:
- Set
TRANSFORMS-<class>=field_extraction
to extract field-value pairs from the log lines at ingest time. - Use
INGEST_EVAL
to add ametric_name
field to every event with agroup
field at ingest time. The newmetric_name
fields get the same values as their correspondinggroup
fields. - Provide separate log-to-metrics settings for the
metric_name=queue
events and themetric_name=pipeline
events. Extract all of the numeric fields from themetric_name=queue
events as measures. - Exclude the
group
,location
, andcorp
fields from the dimensions for themetric_name=queue
metric data points. Exclude thegroup
field from the dimensions for themetric_name=pipeline
events. - Associate the log-to-metrics settings with events that have the
metrics_log
source type.
Those configurations look as follows:
transforms.conf
[eval_pipeline] INGEST_EVAL = metric_name=group [metric-schema:extract_metrics] METRIC-SCHEMA-MEASURES-queue=_ALLNUMS_ METRIC-SCHEMA-BLACKLIST-DIMS-queue=group,location,corp METRIC-SCHEMA-MEASURES-pipeline=cpu_seconds,executes,cumulative_hits METRIC-SCHEMA-BLACKLIST-DIMS-pipeline=group
props.conf
[metrics_log] TRANSFORMS-fieldvalue=field_extraction TRANSFORMS-metricslog=eval_pipeline METRIC-SCHEMA-TRANSFORMS=metric-schema:extract_metrics
The metric data points created by these configurations look like the following examples:
_time | measures | dimensions |
---|---|---|
08-05-2017 20:26:29.073 -0700 | metric_name:queue.max_size_kb=0, metric_name:queue.current_size_kb=0, metric_name:queue.current_size=0, metric_name:queue.largest_size=0, metric_name:queue.smallest_size=0 | name=udp_queue |
08-05-2017 20:26:29.073 -0700 | metric_name:queue.max_size_kb=1024, metric_name:queue.current_size_kb=1, metric_name:queue.current_size=5, metric_name:queue.largest_size=35, metric_name:queue.smallest_size=0 | name=aggqueue |
08-05-2017 20:26:29.073 -0700 | metric_name:queue.max_size_kb=500, metric_name:queue.current_size_kb=0, metric_name:queue.current_size=0, metric_name:queue.largest_size=1, metric_name:queue.smallest_size=0 | name=auditqueue |
08-05-2017 20:26:29.075 -0700 | metric_name:pipeline.cpu_seconds=0, metric_name:pipeline.executes=171, metric_name:pipeline.cumulative_hits=2214401 | name=indexerpipe, processor=indexin |
08-05-2017 20:26:29.075 -0700 | metric_name:pipeline.cpu_seconds=0, metric_name:pipeline.executes=171, metric_name:pipeline.cumulative_hits=2214401, | name=indexerpipe, processor=index_thruput |
08-05-2017 20:26:29.075 -0700 | metric_name:pipeline.cpu_seconds=0, metric_name:pipeline.executes=171, metric_name:pipeline.cumulative_hits=2214401, | name=indexerpipe, processor=indexandforward |
Set up ingest-time log-to-metrics conversion in Splunk Web | Visualize metrics in the Analytics Workspace |
This documentation applies to the following versions of Splunk Cloud Platform™: 9.2.2406 (latest FedRAMP release), 8.2.2112, 8.2.2202, 9.0.2205, 8.2.2201, 8.2.2203, 9.0.2208, 9.0.2209, 9.0.2303, 9.0.2305, 9.1.2308, 9.1.2312, 9.2.2403
Feedback submitted, thanks!