Add context to spans with span tags in Splunk APM 🔗
Enrich the context of the spans you send to Splunk APM by adding span tags. Span tags are key-value pairs that provide additional metadata about spans in a trace.
Note
Span tags are key-value pairs added to spans through instrumentation. In OpenTelemetry, these key-value pairs are known as attributes
.
There are two ways to add span tags to your spans:
Instrument your application to create span tags. This option gives you the most flexibility at the per-application level. Scroll to Instrument your application code to add tags to spans in this topic to learn how.
Add span tags as OpenTelemetry attributes to spans when you send data to the Splunk Distribution of OpenTelemetry Collector. This option lets you add span tags to spans in bulk. Scroll to Add span tags with the Splunk Distribution of OpenTelemetry Collector in this topic to learn how.
If you deploy the Splunk Distribution of OpenTelemetry Collector as a gateway to centrally manage data collection from multiple services, you might choose to instrument your application to define some span tags and manage other span tags with the OpenTelemetry Collector.
Follow span tag naming conventions 🔗
Tags provide more value when you construct a simple, dependable metadata model to use them. Define clear tag names to use for all your applications. Because span tag key-value pairs are text strings, you can create a model to fit your specific needs. As a starting point, OpenTelemetry provides a set of semantic conventions you can use with your spans and traces. For more information, see Trace Semantic Conventions on GitHub.
Add tags to spans 🔗
The following sections describe two ways to add tags to your spans:
Begin by considering where to add your span tags.
Deciding where to add a span tag
When you add span tags to spans via application instrumentation, you have the most control at the per-application level. The tradeoff is that it can be more time-consuming to add an attribute via instrumentation for each of your applications.
Adding span tags in a downstream OpenTelemetry Collector can save time, at the cost of granularity. If a tag is applicable to 100% of the data received by the OpenTelemetry Collector in which you are adding it, adding the tag in your OpenTelemetry Collector config file is fastest. But if you need to apply logic to differentiate the spans that receive a tag via the OpenTelemetry Collector, it is likely faster to add the tags at the application level via instrumentation.
For instance, if multiple applications exist on the same host in a K8s deployment, but in different environments (for instance, production
, development
, and staging
), setting the deployment.environment
tag using the instrumentation library allows you to differentiate environments among spans from the same host.
Note that the deployment.environment
span tag is particularly useful because it lets you filter your entire APM experience by deployment environment. To learn more about environments in Splunk APM, see Set up deployment environments in Splunk APM.
Instrument your application code to add tags to spans 🔗
When you add span tags via instrumentation, you can specify tags on a per-application basis.
How you instrument code to create span tags (attributes) depends on the programming language and instrumentation library you’re using.
The following examples show how to create a custom tag for an existing span:
// Splunk Distribution of OpenTelemetry Java
import io.opentelemetry.api.trace.Span;
Span customizedSpan = Span.current();
customizedSpan.setAttribute("my.attribute","value");
# Splunk Distribution of OpenTelemetry Python
from opentelemetry import trace
customizedSpan = trace.get_current_span()
customizedSpan.set_attribute("my.attribute", "value");
// Splunk Distribution of OpenTelemetry JS
const { context, trace } = require('@opentelemetry/api');
// A span must already exist in the context
const customizedSpan = trace.getSpan(context.active());
customizedSpan.setAttribute('my.attribute', 'value');
// SignalFx Instrumentation for .NET
using OpenTracing;
using OpenTracing.Util;
// A scope for the span must already exist
var span = scope.Span;
span.SetTag("some.tag", "some value");
// You can also set global tags using the SIGNALFX_TRACE_GLOBAL_TAGS
// environment variable, which accepts a list of comma-separated key-value
// pairs. For example, key1:val1,key2:val2.
# SignalFx Ruby Tracing Library
require 'signalfx/tracing'
customizedSpan = OpenTracing.active_span
if customizedSpan
customizedSpan.set_tag('some.tag', 'some value')
end
# You can also set global tags using the SIGNALFX_SPAN_TAGS
# environment variable, which accepts a list of comma-separated key-value
# pairs. For example, key1:val1,key2:val2.
<?php
// SignalFx PHP Tracing Library
use SignalFx\GlobalTracer;
$tracer = GlobalTracer::get(); // Will provide the tracer instance used by provided instrumentations
$customizedSpan = $tracer->startActiveSpan('myApplicationLogic')->getSpan();
$customizedSpan->setTag('some.tag', 'some value');
// You can also set global tags using the SIGNALFX_TRACE_GLOBAL_TAGS
// environment variable, which accepts a list of comma-separated key-value
// pairs. For example: key1:val1,key2:val2.
?>
// SignalFx Go Tracing Library
package main
import (
"github.com/signalfx/signalfx-go-tracing/ddtrace/tracer"
)
if span, ok := tracer.SpanFromContext(ctx); ok {
span.SetTag("some.tag", "some value")
}
// You can also set global tags using the SIGNALFX_SPAN_TAGS
// environment variable, which accepts a list of comma-separated key-value
// pairs. For example, key1:val1,key2:val2.
Add span tags with the Splunk Distribution of OpenTelemetry Collector 🔗
To add a span tag to spans received by the Splunk Distribution of OpenTelemetry Collector, you can use the attributes
processor in your OpenTelemetry Collector configuration YAML file. The generic attributes processor is called attributes
, and any subsequent attributes/<NAME>
processors are uniquely named instances of this defined processor.
Follow these steps to define a new attributes processor and add it to your pipeline:
Define an attributes processor that adds your desired span tag. There are two ways to do this:
Use the
insert
action to set a new key-value pair.For instance, the following code sample adds these key-value pair
enduser.role:"admin"
in spans where the keyenduser.role
doesn’t already exist:processors: ... attributes/setenduser.role: actions: - key: enduser.role value: "admin" action: insertUse the
upsert
action to copy a value from an existing key in the spans and add it to a new key, overriding any existing values for that key.For instance, the following code sample copies the value from the existing
myTenant
key to thetenant
key and overrides any existing values for thetenant
key:processors: ... attributes/settenant: actions: - key: tenant from_attribute: myTenant action: upsertAdd the attributes processor you’ve created to the list of processors under
pipelines
. Place it after thebatch
processor and before thequeued_retry
processor, as theattributes/settenant
processor is placed in the following code sample:service: pipelines: traces: receivers: ... processors: [..., batch, attributes/settenant, queued_retry, ...] ...
Where do host-specific span tags come from? 🔗
If you send trace data to the Splunk Distribution of OpenTelemetry Collector, the collector automatically adds a host
span tag to every span to identify which infrastructure component each span uses. The host
span tag value is generally the hostname
or unique resource identifier for the infrastructure component.
The host
span tag allows Splunk APM to render key infrastructure metrics and link to default dashboards for infrastructure components. This can help you more easily monitor the performance of your applications at the infrastructure level and leverage interactions between Splunk APM and Splunk Infrastructure Monitoring.
In addition to the host
span tag, the OpenTelemetry Collector automatically adds certain span tags to every span captured on that host, according to the type of host. These additional span tags provide more information about infrastructure components each span uses, render corresponding infrastructure metrics, and link to more complete dashboards for the underlying infrastructure component.
The following table provides examples of host-specific span tags:
Span tag |
Description |
---|---|
|
Unique resource identifier for cloud providers. |
|
Unique resource identifier for Docker containers. |
|
Unique resource identifier for a resource in a Kubernetes cluster. |