Docs » Get started with the Splunk Distribution of the OpenTelemetry Collector » Collector components » Collector components: Processors » Metrics transform processor

Metrics transform processor 🔗

The Splunk Distribution of the OpenTelemetry Collector uses the Metrics transform processor to perform the following tasks:

  • Rename metrics.

  • Add, rename, or delete label keys and values.

  • Scale and aggregate metrics across labels or label values.

The processor only supports renames and aggregations within a batch of metrics. It doesn’t do any aggregation across batches, so do not use it to aggregate metrics from multiple sources, such as multiple nodes or clients.

For the complete list of supported operations, see Available operations.

Note

Use the Attributes processor to delete, hash or extract the attributes of spans, metrics, or logs through actions. Use cases include obfuscating sensitive information, copying values to new keys, and backfilling attributes. See Attributes processor.

To include or exclude whole spans, logs, or metrics, use the filter processor. See Filter processor.

Get started 🔗

Follow these steps to configure and activate the component:

  1. Deploy the Splunk Distribution of the OpenTelemetry Collector to your host or container platform:

  1. Configure the metricstransform processor as described in the next section.

  2. Restart the Collector.

Sample configuration 🔗

To activate the resource processor, add metricstransform to the processors section of your configuration file.

For example:

processors:
  metrics_transform:

To complete the configuration, include the processor in the metrics pipeline of the service section of your configuration file. For example:

service:
  pipelines:
    metrics:
      processors: [metricstransform]

Configuration example 🔗

To configure the processor you must specify the list of transformations and operations you want to apply to your metrics. Later transformations or operations reference the result of previous transformations or operations.

processors:
  metricstransform:
  # transforms is a list of transformations with each element transforming a metric selected by metric name
    transforms:

        # SPECIFY WHICH METRIC(S) TO MATCH

        # include specifies the metric name used to determine which metric(s) to operate on
      - include: <metric_name>

        # match_type specifies whether the include name should be used as a strict match or regexp match, default = strict
        match_type: {strict, regexp}

        # experimental_match_labels specifies the label set against which the metric filter will work. If experimental_match_labels is specified, transforms will only be applied to those metrics which
        # have the provided metric label values. This works for both strict and regexp match_type. This is an experimental feature.
        experimental_match_labels: {<label1>: <label_value1>, <label2>: <label_value2>}

        # SPECIFY THE ACTION TO TAKE ON THE MATCHED METRIC(S)

        # action specifies if the operations (specified below) are performed on metrics in place (update), on an inserted clone (insert), or on a new combined metric (combine)
        action: {update, insert, combine}

        # SPECIFY HOW TO TRANSFORM THE METRIC GENERATED AS A RESULT OF APPLYING THE ABOVE ACTION

        # new_name specifies the updated name of the metric; if action is insert or combine, new_name is required
        new_name: <new_metric_name_inserted>

        # aggregation_type defines how combined data points will be aggregated; if action is combine, aggregation_type is required
        aggregation_type: {sum, mean, min, max}

        # submatch_case specifies the case that should be used when adding label values based on regexp submatches when performing a combine action; leave blank to use the submatch value as is
        submatch_case: {lower, upper}

        # operations contain a list of operations that will be performed on the resulting metric(s)
        operations:
            # action defines the type of operation that will be performed, see examples below for more details
          - action: {add_label, update_label, delete_label_value, toggle_scalar_data_type, experimental_scale_value, aggregate_labels, aggregate_label_values}
            # label specifies the label to operate on
            label: <label>
            # new_label specifies the updated name of the label; if action is add_label, new_label is required
            new_label: <new_label>
            # aggregated_values contains a list of label values that will be aggregated; if action is aggregate_label_values, aggregated_values is required
            aggregated_values: [values...]
            # new_value specifies the updated name of the label value; if action is add_label or aggregate_label_values, new_value is required
            new_value: <new_value>
            # label_value specifies the label value for which points should be deleted; if action is delete_label_value, label_value is required
            label_value: <label_value>
            # label_set contains a list of labels that will remain after aggregation; if action is aggregate_labels, label_set is required
            label_set: [labels...]
            # aggregation_type defines how data points will be aggregated; if action is aggregate_labels or aggregate_label_values, aggregation_type is required
            aggregation_type: {sum, mean, min, max}
            # experimental_scale specifies the scalar to apply to values
            experimental_scale: <scalar>
            # value_actions contain a list of operations that will be performed on the selected label
            value_actions:
                # value specifies the value to operate on
              - value: <current_label_value>
                # new_value specifies the updated value
                new_value: <new_label_value>

See Available operations to understand the operations, and see some examples.

Available operations 🔗

The processor can perform the following operations:

  • Rename metrics. For example, rename system.cpu.usage``to ``system.cpu.usage_time.

  • Add labels. For example, you can add the new label identifier with value 1 to all points.

  • Rename label keys. For example, rename the label state to cpu_state.

  • Rename label values. For example, in the label state, you can rename the value idle to -.

  • Delete data points. For example, delete all points where the label state has value idle.

  • Toggle data type. For example, you can change int data points to double data points

  • Scale value. For example, multiply values by 1000 to convert from seconds to milliseconds.

  • Aggregate across label sets. For example, you can retain only the label state, and average all points with the same value for this label.

  • Aggregate across label values. For example, for the label state, you can sum points where the value is user or system into used = user + system.

The following applies:

  • You can only apply operations to one or more metrics using a strict or regexp filter.

  • With the action property you can:

    • Update your metrics in-place (update).

    • Copy and update the copied metrics (insert).

    • Combine your metrics into a newly inserted metric that is generated by combining all data points from the set of matching metrics into a single metric (combine). The original matching metrics are also removed.

  • When renaming metrics, capturing groups from the regexp filter will be expanded.

  • When adding or updating a label value, {{version}} will be replaced with your Collector’s instance version number.

Example: Create a new metric from an existing metric 🔗

To create a new metric from an existing metric, apply this configuration:

# create host.cpu.utilization from host.cpu.usage
include: host.cpu.usage
action: insert
new_name: host.cpu.utilization
operations:
  ...

Example: Create a new metric from an existing metric with matching label values 🔗

To create a new metric from an existing metric with matching label values, apply this configuration:

# create host.cpu.utilization from host.cpu.usage where we have metric label "container=my_container"
include: host.cpu.usage
action: insert
new_name: host.cpu.utilization
match_type: strict
experimental_match_labels: {"container": "my_container"}
operations:
  ...

Example: Create a new metric from an existing metric with matching label values with regexp 🔗

To create a new metric from an existing metric with matching label values with regexp, apply this configuration:

# create host.cpu.utilization from host.cpu.usage where we have metric label pod with non-empty values
include: host.cpu.usage
action: insert
new_name: host.cpu.utilization
match_type: regexp
experimental_match_labels: {"pod": "(.|\\s)*\\S(.|\\s)*"}
operations:
  ...

Example: Rename a metric 🔗

To rename a metric, apply this configuration:

# rename system.cpu.usage to system.cpu.usage_time
include: system.cpu.usage
action: update
new_name: system.cpu.usage_time

Example: Rename multiple metrics using Substitution 🔗

To rename multiple metrics using Substitution, apply this configuration:

# rename all system.cpu metrics to system.processor.*.stat
# instead of regular $ use double dollar $$. Because $ is treated as a special character.
# wrap the group name/number with braces
include: ^system\.cpu\.(.*)$$
match_type: regexp
action: update
new_name: system.processor.$${1}.stat

Example: Add a label 🔗

To add a label, apply this configuration:

# for system.cpu.usage_time, add label `version` with value `opentelemetry collector vX.Y.Z` to all points
include: system.cpu.usage
action: update
operations:
  - action: add_label
    new_label: version
    new_value: opentelemetry collector {{version}}

Example: Add a label to multiple metrics 🔗

To add a label to multiple metrics, apply this configuration:

# for all system metrics, add label `version` with value `opentelemetry collector vX.Y.Z` to all points
include: ^system\.
match_type: regexp
action: update
operations:
  - action: add_label
    new_label: version
    new_value: opentelemetry collector {{version}}

Example: Rename labels 🔗

To rename labels, apply this configuration:

# for system.cpu.usage_time, rename the label state to cpu_state
include: system.cpu.usage
action: update
operations:
  - action: update_label
    label: state
    new_label: cpu_state

Example: Rename labels for multiple metrics 🔗

To rename labels for multiple metrics, apply this configuration:

# for all system.cpu metrics, rename the label state to cpu_state
include: ^system\.cpu\.
action: update
operations:
  - action: update_label
    label: state
    new_label: cpu_state

Example: Rename label values 🔗

To rename label values, apply this configuration:

# rename the label value slab_reclaimable to sreclaimable, slab_unreclaimable to sunreclaimable
include: system.memory.usage
action: update
operations:
  - action: update_label
    label: state
    value_actions:
      - value: slab_reclaimable
        new_value: sreclaimable
      - value: slab_unreclaimable
        new_value: sunreclaimable

Example: Delete by label value 🔗

To delete by label value, apply this configuration:

# deletes all data points with the label value 'idle' of the label 'state'
include: system.cpu.usage
action: update
operations:
  - action: delete_label_value
    label: state
    label_value: idle

Example: Toggle the data type 🔗

To toggle the data type, apply this configuration:

# toggle the datatype of cpu usage from int (the default) to double
include: system.cpu.usage
action: update
operations:
  - action: toggle_scalar_data_type

Example: Scale values 🔗

To scale values, apply this configuration:

# experimental_scale CPU usage from seconds to milliseconds
include: system.cpu.usage
action: update
operations:
  - action: experimental_scale_value
    experimental_scale: 1000

Example: Aggregate labels 🔗

To aggregate labels, apply this configuration:

# aggregate away all labels except `state` using summation
include: system.cpu.usage
action: update
operations:
  - action: aggregate_labels
    label_set: [ state ]
    aggregation_type: sum

Example: Aggregate label values 🔗

To aggregate label values, apply this configuration:

# aggregate data points with state label value slab_reclaimable & slab_unreclaimable using summation into slab
include: system.memory.usage
action: update
operations:
  - action: aggregate_label_values
    label: state
    aggregated_values: [ slab_reclaimable, slab_unreclaimable ]
    new_value: slab
    aggregation_type: sum

Example: Combine metrics 🔗

To combine metrics, apply this configuration:

# convert a set of metrics for each http_method into a single metric with an http_method label, i.e.
#
# Web Service (*)/Total Delete Requests     iis.requests{http_method=delete}
# Web Service (*)/Total Get Requests     >  iis.requests{http_method=get}
# Web Service (*)/Total Post Requests       iis.requests{http_method=post}
include: ^Web Service \(\*\)/Total (?P<http_method>.*) Requests$
match_type: regexp
action: combine
new_name: iis.requests
submatch_case: lower
operations:
  ...

Example: Group Metrics 🔗

To group Metrics, apply this configuration:

# Group metrics from one single ResourceMetrics and report them as multiple ResourceMetrics.
#
# ex: Consider pod and container metrics collected from Kubernetes. Both the metrics are recorded under under one ResourceMetric
# applying this transformation will result in two separate ResourceMetric packets with corresponding resource labels in the resource headers
#
# instead of regular $ use double dollar $$. Because $ is treated as a special character.

- include: ^k8s\.pod\.(.*)$$
  match_type: regexp
  action: group
  group_resource_labels: {"resouce.type": "k8s.pod", "source": "kubelet"}
- include: ^container\.(.*)$$
  match_type: regexp
  action: group
  group_resource_labels: {"resouce.type": "container", "source": "kubelet"}

Settings 🔗

The following table shows the configuration options for the metricstransform processor:

Troubleshooting 🔗

If you are a Splunk Observability Cloud customer and are not able to see your data in Splunk Observability Cloud, you can get help in the following ways.

Available to Splunk Observability Cloud customers

Available to prospective customers and free trial users

  • Ask a question and get answers through community support at Splunk Answers .

  • Join the Splunk #observability user group Slack channel to communicate with customers, partners, and Splunk employees worldwide. To join, see Chat groups in the Get Started with Splunk Community manual.

To learn about even more support options, see Splunk Customer Success .