Control data to ingest using the Collector 🔗
Depending on its configuration, the Splunk Distribution of OpenTelemetry Collector can forward a wide range of telemetry, such as metrics, traces, or logs, to the Observability Cloud ingest endpoints. For certain use cases, some of this data can be redundant, unnecessary, or sensitive, causing technical complications, increased cost, or legal issues.
To address these situations, the Collector comes with options to process the data you’re receiving to modify or delete unwanted elements before they’re ingested by Observability Cloud. For example, you can use the attributes processor to edit or remove any unwanted data.
Use case: Remove dimensions using the attributes processor 🔗
Moira, a performance engineer, notices high cardinality in ingested metrics, which heavily impacts data charges. Moira is considering dropping certain dimensions to cut costs.
Moira checks which dimensions are being used and realizes that for the metric
cpu.utilization, the dimensions
source_host are irrelevant. They decide that those dimension don’t need to be ingested at all.
To prevent both dimensions from being ingested, first Moira adds the attributes processor in the Collectors’s configuration, set up to skip the unnecessary dimensions:
extensions: ... processors: attributes/delete: actions: - key: hostname action: delete - key: source_host action: delete service: ... ...
Next, Moira adds the
attributes/delete processor to the
processors pipeline under
pipelines in the Collector’s configuration:
... service: pipelines: traces: receivers: ... processors: [..., attributes/delete, ...] ...
Alternatives to alter or remove data 🔗
See another example of how to tweak your data at Remove sensitive data using the Splunk Distribution of OpenTelemetry Collector.
You can also use the metrics pipeline management tool to control how you ingest and store your metrics. Learn more at Introduction to metrics pipeline management.