Docs » Analyze services with span tags and MetricSets in Splunk APM » Add context to spans with span tags in Splunk APM

Add context to spans with span tags in Splunk APM 🔗

Enrich the context of the spans you send to Splunk APM by adding span tags. Span tags are key-value pairs that provide additional metadata about spans in a trace. In OpenTelemetry, span tags are known as attributes.

There are two ways to add span tags to your spans:

If you deploy the Splunk Distribution of OpenTelemetry Collector as a gateway to centrally manage data collection from multiple services, you might want to instrument your application to define span tags and manage other span tags with the Collector.

Follow span tag naming conventions 🔗

Tags provide more value when you use a naming convention. Define clear tag names to use for all your applications. Because span tag key-value pairs are text strings, you can create a model to fit your specific needs.

As a starting point, OpenTelemetry provides a set of semantic conventions you can use with your spans and traces. For more information, see OpenTelemetry Trace semantic conventions on GitHub.

Add tags to spans 🔗

The following sections describe two ways to add tags to your spans:

Begin by considering where to add your span tags.

Note

Deciding where to add a span tag

When you add span tags to spans via application instrumentation, you have the most control at the per-application level. The tradeoff is that it can be more time-consuming to add an attribute via instrumentation for each of your applications.

Adding span tags in a downstream OpenTelemetry Collector can save time, at the cost of granularity. If a tag is applicable to 100% of the data received by the OpenTelemetry Collector in which you are adding it, adding the tag in your OpenTelemetry Collector config file is fastest. But if you need to apply logic to differentiate the spans that receive a tag via the OpenTelemetry Collector, it is likely faster to add the tags at the application level via instrumentation.

For instance, if multiple applications exist on the same host in a K8s deployment, but in different environments (for instance, production, development, and staging), setting the deployment.environment tag using the instrumentation library allows you to differentiate environments among spans from the same host.

Note that the deployment.environment span tag is particularly useful, as it lets you filter your entire APM experience by deployment environment. To learn more about environments in Splunk APM, see Set up deployment environments in Splunk APM.

Instrument your application code to add tags to spans 🔗

When you add span tags, you can specify tags on a per-application basis. How you instrument code to create span tags or attributes depends on the programming language and instrumentation library you’re using.

The following examples show how to create a custom tag for an existing span:

// Splunk Distribution of OpenTelemetry Java

import io.opentelemetry.api.trace.Span;

Span customizedSpan = Span.current();

customizedSpan.setAttribute("my.attribute","value");

// You can also set global tags using the OTEL_RESOURCE_ATTRIBUTES
// environment variable, which accepts a list of comma-separated key-value
// pairs. For example, key1:val1,key2:val2.

Add span tags with the Splunk Distribution of OpenTelemetry Collector 🔗

To add a span tag to spans received by the Splunk Distribution of OpenTelemetry Collector, you can use the attributes processor in your OpenTelemetry Collector configuration YAML file.

The generic attributes processor is called attributes. Any subsequent attributes/<NAME> processors are named instances of the attributes processor. See Attributes processor for more information.

Follow these steps to define a new attributes processor and add it to your pipeline:

  1. Define an attributes processor that adds your desired span tag. There are two ways to do this:

    • Use the insert action to set a new key-value pair.

      For instance, the following code sample adds these key-value pair enduser.role:"admin" in spans where the key enduser.role doesn’t already exist:

      processors:
  ...
  attributes/setenduser.role:
    actions:
    - key: enduser.role
        value: "admin"
        action: insert

    • Use the upsert action to copy a value from an existing key in the spans and add it to a new key, overriding any existing values for that key.

      For instance, the following code sample copies the value from the existing myTenant key to the tenant key and overrides any existing values for the tenant key:

      processors:
  ...
  attributes/settenant:
    actions:
    - key: tenant
        from_attribute: myTenant
        action: upsert

  2. Add the attributes processor you’ve created to the list of processors under pipelines. Place it after the batch processor and before the queued_retry processor, as the attributes/settenant processor is placed in the following code sample:

    service:
  pipelines:
    traces:
      receivers: ...
      processors: [...,  batch, attributes/settenant,  queued_retry, ...]
      ...

Where do host-specific span tags come from? 🔗

The Splunk Distribution of OpenTelemetry Collector automatically adds a host span tag to every span to identify which infrastructure component each span uses. The host span tag value is generally the hostname or unique resource identifier for the infrastructure component.

The host span tag allows Splunk APM to render key infrastructure metrics and link to default dashboards for infrastructure components. This can help you more easily monitor the performance of your applications at the infrastructure level and leverage interactions between Splunk APM and Splunk Infrastructure Monitoring.

In addition to the host span tag, the OpenTelemetry Collector automatically adds certain span tags to every span captured on that host, according to the type of host. These additional span tags provide more information about infrastructure components each span uses, render corresponding infrastructure metrics, and link to more complete dashboards for the underlying infrastructure component.

The following table provides examples of host-specific span tags:

Span tag

Description

AWSUniqueId or gcp_id

Unique resource identifier for cloud providers.

container_id

Unique resource identifier for Docker containers.

kubernetes_pod_uid

Unique resource identifier for a resource in a Kubernetes cluster.