Filter processor 🔗

The filter processor is an OpenTelemetry Collector component that filters spans, metrics, or logs based on the conditions you define in its configuration. A typical use case for the filter processor is dropping telemetry that isn’t relevant to the observed system, like noncritical logs or spans, to reduce noise in your data.

Filtering works through allow lists and deny lists, which include or exclude telemetry based on regular expressions and resource attributes. You can also use the OpenTelemetry Transformation Language (OTTL) to better describe the signals you want to filter. The processor supports all pipeline types. See Process your data with pipelines for more information.

The filter processor can include or exclude telemetry based on the following criteria:

Signal	Criteria and match types
Spans	OTTL conditions, span names (`strict` or `regexp`), and resource attributes (`strict` or `regexp`). Span events filtering only supports OTTL conditions.
Metrics	OTTL conditions, metric names (`strict` or `regexp`), and metric attributes (`expr`). Data points filtering only supports OTTL conditions.
Logs	OTTL conditions, resource attributes (`strict` or `regexp`).

The regular expression engine used by the filter processor is re2.

Note

To redact or transform attributes of spans, logs, or metrics without dropping them, use the attributes processor. See Attributes processor.

Get started 🔗

By default, the Splunk Distribution of OpenTelemetry Collector includes the filter processor. To activate the filter processor for a pipeline, add filter to the processors section of the configuration. For example:

processors:
  filter/includemetrics:
    metrics:
        # Drop nonmatching metrics from the pipeline
        include:
          match_type: strict
          metric_names:
            - good_metric
            - great_metric
  filter/excludemetrics:
    metrics:
      # Drop matching metrics from the pipeline
      exclude:
        match_type: strict
        metric_names:
          - a_metric
          - another_metric
          - a_third_metric
  filter/mixedlogs:
    logs:
       # Include filters are applied before exclude filters
       include:
         match_type: strict
         record_attributes:
           - key: host.name
             value: "(host1|anotherhost2)"
       exclude:
         match_type: strict
         record_attributes:
           - key: host.name
             value: wrong_host_.*

You can then add the filter processors to any compatible pipeline. For example:

:emphasize-lines: 6, 14, 15, 23

service:
  pipelines:
    traces:
      receivers: [jaeger, otlp, zipkin]
      processors:
      - filter/traces
      - memory_limiter
      - batch
      - resourcedetection
      exporters: [sapm, signalfx]
    metrics:
      receivers: [hostmetrics, otlp, signalfx]
      processors:
      - filter/includemetrics
      - filter/excludemetrics
      - memory_limiter
      - batch
      - resourcedetection
      exporters: [signalfx]
    logs:
      receivers: [fluentforward, otlp]
      processors:
      - filter/mixedlogs
      - memory_limiter
      - batch
      - resourcedetection
      exporters: [splunk_hec]

For a complete list of parameters, see Settings.

Sample configurations 🔗

The following sample configurations show how to filter spans, metrics, and logs using different criteria.

Note

For a complete list of examples, see the configuration snippets in https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/processor/filterprocessor/testdata.

Filter spans 🔗

You can exclude or include spans from traces using resource attributes or OTTL conditions. For example:

filter/spans:
  spans:
    include:
      match_type: strict
      services:
        - ponygame
        - ponytest
      attributes:
        - key: an_attribute
          value: "(valid_value|another_value)"
    exclude:
      match_type: regexp
      attributes:
        - key: bad_attributes
          value: "(invalid_value|another_value)"

filter/ottl:
  traces:
    span:
      - 'attributes["test"] == "value"'
      - 'attributes["test"] == "value2"'

Note

Include filters are always applied before exclude filters for any given filter processor instance.

Filter metrics 🔗

You can exclude or include metrics using metric names, expressions, or OTTL conditions. For example:

filter/mixed:
  metrics:
    # Include using metric names
    include:
      match_type: strict
      metric_names:
        - a_metric
        - another_metric
    # Exclude using regular expressions
    exclude:
      match_type: regexp
      metric_names:
        - prefix/.*
        - prefix_.*
        - .*/suffix
        - .*_suffix
        - .*/contains/.*
        - .*_contains_.*
        - full/name/match
        - full_name_match

filter/expr:
  metrics:
    include:
      match_type: expr
      expressions:
        - Label("label1") == "text"
        - HasLabel("label2")

filter/ottl:
  metrics:
    metric:
      - 'name == "a_name"'
   datapoint:
      - 'attributes["attributename"] == "value"'

Filter logs 🔗

You can exclude or include logs using resource attributes or OTTL conditions. For example:

filter/mixed:
  logs:
    include:
      match_type: strict
      resource_attributes:
        - key: should_include
          value: "true"
    exclude:
      match_type: regexp
      resource_attributes:
        - key: host.name
          value: banned_host_.*

filter/severity:
  logs:
    exclude:
      match_type: strict
      severity_texts:
        - "DEBUG"
        - "DEBUG2"
        - "DEBUG3"
        - "DEBUG4"
    include:
      match_type: regexp
      severity_texts:
        - "INFO[2-4]?"

filter/recordattributes:
  logs:
    exclude:
      match_type: strict
      record_attributes:
        - key: should_exclude
          value: "true"

filter/includeexclude:
  logs:
    include:
      severity_number:
        min: "INFO"
        match_undefined: true
   exclude:
      severity_number:
        min: "ERROR"

filter/ottl:
  logs:
    log_record:
      - 'attributes["test"] == "pass"'

Filter Kubernetes elements 🔗

You can exclude or include Kubernetes elements, such as containers, pods, nodes, namespaces, or clusters, with the following configuration:

agent:
  config:
    processors:
      # Exclude specific metrics from containers named 'containerXName' or 'containerYName'
      filter/exclude_metrics_from_container:
        metrics:
          exclude:
            match_type: regexp
            resource_attributes:
              - key: k8s.container.name
                value: '^(containerXName|containerYName)$'

      # Exclude logs from pods named 'podNameX'
      filter/exclude_logs_from_pod:
        logs:
          exclude:
            match_type: regexp
            resource_attributes:
              - key: k8s.pod.name
                value: '^(podNameX)$'

      # Exclude logs from nodes named 'nodeNameX'
      filter/exclude_logs_from_node:
        logs:
          exclude:
            match_type: regexp
            resource_attributes:
              - key: k8s.node.name
                value: '^(nodeNameX)$'

      # Exclude spans from traces for services housed in containers named 'containerXName' or 'containerYName'
      filter/exclude_spans_from_traces_from_container:
        spans:
          exclude:
            match_type: regexp
            attributes:
              - key: k8s.container.name
                value: '^(containerXName|containerYName)$'

      # Exclude all telemetry data (metrics, logs, traces) from a namespace named 'namespaceX'
      filter/exclude_all_telemetry_data_from_namespace:
        logs:
          exclude:
            match_type: regexp
            resource_attributes:
              - key: k8s.namespace.name
                value: '^(namespaceX)$'
        metrics:
          exclude:
            match_type: regexp
            resource_attributes:
              - key: k8s.namespace.name
                value: '^(namespaceX)$'
        traces:
          span:
            - 'attributes["k8s.namespace.name"] != "namespaceX"'

      # Exclude metrics from a cluster named 'clusterX'
      filter/exclude_metrics_from_cluster:
        metrics:
          exclude:
            match_type: regexp
            resource_attributes:
              - key: k8s.cluster.name
                value: '^(clusterX)$'

After setting up the processors, configure the pipelines:

# Define the data processing pipelines for logs, metrics, and traces
service:
  pipelines:
    logs:
      processors:
        - memory_limiter
        - k8sattributes
        - filter/logs
        - batch
        - resourcedetection
        - resource
        - resource/logs
        - filter/exclude_logs_from_pod
        - filter/exclude_logs_from_node

Drop telemetry using OTTL conditions 🔗

You can use the OpenTelemetry Transformation Language (OTTL) to define filtering conditions with more detail. Matching telemetry is dropped (excluded).

In OTTL, each telemetry type or context has its own fields. The following example shows all available OTTL contexts:

processors:
  filter:
     traces:
       span:
         - 'attributes["attribute.label"] == "attribute_value"'
         - 'resource.attributes["host.name"] == "localhost"'
       # Checked only if `span` is not dropped
       spanevent:
         - 'attributes["label"] == true'
         - 'IsMatch(name, ".*http.*") == false'
        # If all span events are dropped, the span is dropped
       metrics:
         metric:
           - 'name == "metric.name" and attributes["label"] == "value"'
           - 'type == METRIC_DATA_TYPE_HISTOGRAM'
         # Checked only if `metric` is not dropped
         datapoint:
           - 'metric.type == METRIC_DATA_TYPE_SUMMARY'
           - 'resource.attributes["service.name"] == "my_service_name"'
          # If all datapoints are dropped, the metric is dropped
       logs:
         log_record:
           - 'IsMatch(body, ".*token.*") == true'
           - 'severity_number < SEVERITY_NUMBER_WARN'

For more information on OTTL functions and syntax, see:

Settings 🔗

The following table shows the configuration options for the filter processor:

Troubleshooting 🔗

If you are a Splunk Observability Cloud customer and are not able to see your data in Splunk Observability Cloud, you can get help in the following ways.

Available to Splunk Observability Cloud customers

Submit a case in the Splunk Support Portal .
Contact Splunk Support .

Available to prospective customers and free trial users

Ask a question and get answers through community support at Splunk Answers .
Join the Splunk #observability user group Slack channel to communicate with customers, partners, and Splunk employees worldwide. To join, see Chat groups in the Get Started with Splunk Community manual.

This page was last updated on Oct 10, 2024.

Related Topics

Filter processor 🔗

Get started 🔗

Sample configurations 🔗

Filter spans 🔗

Filter metrics 🔗

Filter logs 🔗

Filter Kubernetes elements 🔗

Drop telemetry using OTTL conditions 🔗

Settings 🔗

Troubleshooting 🔗

Was this topic useful?

Splunk

Related Topics