Filter processor π
The filter processor is an OpenTelemetry Collector component that filters spans, metrics, or logs based on the conditions you define in its configuration. A typical use case for the filter processor is dropping telemetry that isnβt relevant to the observed system, like noncritical logs or spans, to reduce noise in your data.
Filtering works through allow lists and deny lists, which include or exclude telemetry based on regular expressions and resource attributes. You can also use the OpenTelemetry Transformation Language (OTTL) to better describe the signals you want to filter. The processor supports all pipeline types. See Process your data with pipelines for more information.
The filter processor can include or exclude telemetry based on the following criteria:
Signal |
Criteria and match types |
---|---|
Spans |
OTTL conditions, span names ( |
Metrics |
OTTL conditions, metric names ( |
Logs |
OTTL conditions, resource attributes ( |
The regular expression engine used by the filter processor is re2
.
Note
To redact or transform attributes of spans, logs, or metrics without dropping them, use the attributes processor. See Attributes processor.
Get started π
By default, the Splunk Distribution of OpenTelemetry Collector includes the filter processor. To activate the filter processor for a pipeline, add filter
to the processors
section of the configuration. For example:
processors:
filter/includemetrics:
metrics:
# Drop nonmatching metrics from the pipeline
include:
match_type: strict
metric_names:
- good_metric
- great_metric
filter/excludemetrics:
metrics:
# Drop matching metrics from the pipeline
exclude:
match_type: strict
metric_names:
- a_metric
- another_metric
- a_third_metric
filter/mixedlogs:
logs:
# Include filters are applied before exclude filters
include:
match_type: strict
record_attributes:
- key: host.name
value: "(host1|anotherhost2)"
exclude:
match_type: strict
record_attributes:
- key: host.name
value: wrong_host_.*
You can then add the filter processors to any compatible pipeline. For example:
:emphasize-lines: 6, 14, 15, 23
service:
pipelines:
traces:
receivers: [jaeger, otlp, zipkin]
processors:
- filter/traces
- memory_limiter
- batch
- resourcedetection
exporters: [sapm, signalfx]
metrics:
receivers: [hostmetrics, otlp, signalfx]
processors:
- filter/includemetrics
- filter/excludemetrics
- memory_limiter
- batch
- resourcedetection
exporters: [signalfx]
logs:
receivers: [fluentforward, otlp]
processors:
- filter/mixedlogs
- memory_limiter
- batch
- resourcedetection
exporters: [splunk_hec]
For a complete list of parameters, see Settings.
Sample configurations π
The following sample configurations show how to filter spans, metrics, and logs using different criteria.
Note
For a complete list of examples, see the configuration snippets in https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/processor/filterprocessor/testdata.
Filter spans π
You can exclude or include spans from traces using resource attributes or OTTL conditions. For example:
filter/spans:
spans:
include:
match_type: strict
services:
- ponygame
- ponytest
attributes:
- key: an_attribute
value: "(valid_value|another_value)"
exclude:
match_type: regexp
attributes:
- key: bad_attributes
value: "(invalid_value|another_value)"
filter/ottl:
traces:
span:
- 'attributes["test"] == "value"'
- 'attributes["test"] == "value2"'
Note
Include filters are always applied before exclude filters for any given filter processor instance.
Filter metrics π
You can exclude or include metrics using metric names, expressions, or OTTL conditions. For example:
filter/mixed:
metrics:
# Include using metric names
include:
match_type: strict
metric_names:
- a_metric
- another_metric
# Exclude using regular expressions
exclude:
match_type: regexp
metric_names:
- prefix/.*
- prefix_.*
- .*/suffix
- .*_suffix
- .*/contains/.*
- .*_contains_.*
- full/name/match
- full_name_match
filter/expr:
metrics:
include:
match_type: expr
expressions:
- Label("label1") == "text"
- HasLabel("label2")
filter/ottl:
metrics:
metric:
- 'name == "a_name"'
datapoint:
- 'attributes["attributename"] == "value"'
Filter logs π
You can exclude or include logs using resource attributes or OTTL conditions. For example:
filter/mixed:
logs:
include:
match_type: strict
resource_attributes:
- key: should_include
value: "true"
exclude:
match_type: regexp
resource_attributes:
- key: host.name
value: banned_host_.*
filter/severity:
logs:
exclude:
match_type: strict
severity_texts:
- "DEBUG"
- "DEBUG2"
- "DEBUG3"
- "DEBUG4"
include:
match_type: regexp
severity_texts:
- "INFO[2-4]?"
filter/recordattributes:
logs:
exclude:
match_type: strict
record_attributes:
- key: should_exclude
value: "true"
filter/includeexclude:
logs:
include:
severity_number:
min: "INFO"
match_undefined: true
exclude:
severity_number:
min: "ERROR"
filter/ottl:
logs:
log_record:
- 'attributes["test"] == "pass"'
Filter Kubernetes elements π
You can exclude or include Kubernetes elements, such as containers, pods, nodes, namespaces, or clusters, with the following configuration:
agent:
config:
processors:
# Exclude specific metrics from containers named 'containerXName' or 'containerYName'
filter/exclude_metrics_from_container:
metrics:
exclude:
match_type: regexp
resource_attributes:
- key: k8s.container.name
value: '^(containerXName|containerYName)$'
# Exclude logs from pods named 'podNameX'
filter/exclude_logs_from_pod:
logs:
exclude:
match_type: regexp
resource_attributes:
- key: k8s.pod.name
value: '^(podNameX)$'
# Exclude logs from nodes named 'nodeNameX'
filter/exclude_logs_from_node:
logs:
exclude:
match_type: regexp
resource_attributes:
- key: k8s.node.name
value: '^(nodeNameX)$'
# Exclude spans from traces for services housed in containers named 'containerXName' or 'containerYName'
filter/exclude_spans_from_traces_from_container:
spans:
exclude:
match_type: regexp
attributes:
- key: k8s.container.name
value: '^(containerXName|containerYName)$'
# Exclude all telemetry data (metrics, logs, traces) from a namespace named 'namespaceX'
filter/exclude_all_telemetry_data_from_namespace:
logs:
exclude:
match_type: regexp
resource_attributes:
- key: k8s.namespace.name
value: '^(namespaceX)$'
metrics:
exclude:
match_type: regexp
resource_attributes:
- key: k8s.namespace.name
value: '^(namespaceX)$'
traces:
span:
- 'attributes["k8s.namespace.name"] != "namespaceX"'
# Exclude metrics from a cluster named 'clusterX'
filter/exclude_metrics_from_cluster:
metrics:
exclude:
match_type: regexp
resource_attributes:
- key: k8s.cluster.name
value: '^(clusterX)$'
After setting up the processors, configure the pipelines:
# Define the data processing pipelines for logs, metrics, and traces
service:
pipelines:
logs:
processors:
- memory_limiter
- k8sattributes
- filter/logs
- batch
- resourcedetection
- resource
- resource/logs
- filter/exclude_logs_from_pod
- filter/exclude_logs_from_node
Drop telemetry using OTTL conditions π
You can use the OpenTelemetry Transformation Language (OTTL) to define filtering conditions with more detail. Matching telemetry is dropped (excluded).
In OTTL, each telemetry type or context has its own fields. The following example shows all available OTTL contexts:
processors:
filter:
traces:
span:
- 'attributes["attribute.label"] == "attribute_value"'
- 'resource.attributes["host.name"] == "localhost"'
# Checked only if `span` is not dropped
spanevent:
- 'attributes["label"] == true'
- 'IsMatch(name, ".*http.*") == false'
# If all span events are dropped, the span is dropped
metrics:
metric:
- 'name == "metric.name" and attributes["label"] == "value"'
- 'type == METRIC_DATA_TYPE_HISTOGRAM'
# Checked only if `metric` is not dropped
datapoint:
- 'metric.type == METRIC_DATA_TYPE_SUMMARY'
- 'resource.attributes["service.name"] == "my_service_name"'
# If all datapoints are dropped, the metric is dropped
logs:
log_record:
- 'IsMatch(body, ".*token.*") == true'
- 'severity_number < SEVERITY_NUMBER_WARN'
For more information on OTTL functions and syntax, see:
OTTL Syntax: https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/pkg/ottl/README.md
OTTL Functions: https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/pkg/ottl/ottlfuncs
OTTL Contexts: https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/pkg/ottl/contexts
Settings π
The following table shows the configuration options for the filter processor:
Troubleshooting π
If you are a Splunk Observability Cloud customer and are not able to see your data in Splunk Observability Cloud, you can get help in the following ways.
Available to Splunk Observability Cloud customers
Submit a case in the Splunk Support Portal .
Contact Splunk Support .
Available to prospective customers and free trial users
Ask a question and get answers through community support at Splunk Answers .
Join the Splunk #observability user group Slack channel to communicate with customers, partners, and Splunk employees worldwide. To join, see Chat groups in the Get Started with Splunk Community manual.