Docs » Analyze services with span tags in Splunk APM » Add context to spans with span tags in Splunk APM

Add context to spans with span tags in Splunk APM 🔗

Enrich the context of the spans you send to Splunk APM by adding span tags. Span tags are key-value pairs that provide additional metadata about spans in a trace.

Note

Span tags are key-value pairs added to spans through instrumentation. In OpenTelemetry, these key-value pairs are known as attributes.

There are two ways to add span tags to your spans:

If you deploy the Splunk Distribution of OpenTelemetry Collector as a gateway to centrally manage data collection from multiple services, you might choose to instrument your application to define some span tags and manage other span tags with the OpenTelemetry Collector. For further guidance on deciding where in your pipeline to add span tags, see Deciding where to add a span tag.

Tip

The deployment.environment span tag is particularly useful because it lets you filter your entire APM experience by deployment environment. To learn more about environments in Splunk APM, see Deployment environments.

Follow span tag naming conventions 🔗

Tags provide more value when you construct a simple, dependable metadata model to use them. Define clear tag names to use for all your applications. Because span tag key-value pairs are text strings, you can create a model to fit your specific needs. OpenTelemetry provides semantic conventions for spans that you can use.

Add tags to spans 🔗

The following sections describe two ways to add tags to your spans.

Instrument your application code to add tags to spans 🔗

When you add span tags via instrumentation, you can specify tags on a per-application basis.

How you instrument code to create span tags (attributes) depends on the programming language and instrumentation library you’re using.

The following examples show how to create a custom tag for an existing span:

// Splunk Distribution of OpenTelemetry Java

import io.opentelemetry.api.trace.Span;

Span customizedSpan = Span.current();

customizedSpan.setAttribute("my.attribute","value");

Add span tags with the Splunk Distribution of OpenTelemetry Collector 🔗

To add a span tag to spans received by the Splunk Distribution of OpenTelemetry Collector, you can use the attributes processor in your OpenTelemetry Collector configuration YAML file. The generic attributes processor is called attributes, and any subsequent attributes/<NAME> processors are uniquely named instances of this defined processor.

Follow these steps to define a new attributes processor and add it to your pipeline:

  1. Define an attributes processor that adds your desired span tag. There are two ways to do this:

    • Use the insert action to set a new key-value pair.

      For instance, the following code sample adds these key-value pair enduser.role:"admin" in spans where the key enduser.role doesn’t already exist:

      processors:
        ...
        attributes/setenduser.role:
          actions:
          - key: enduser.role
              value: "admin"
              action: insert
      
    • Use the upsert action to copy a value from an existing key in the spans and add it to a new key, overriding any existing values for that key.

      For instance, the following code sample copies the value from the existing myTenant key to the tenant key and overrides any existing values for the tenant key:

      processors:
        ...
        attributes/settenant:
          actions:
          - key: tenant
              from_attribute: myTenant
              action: upsert
      
  2. Add the attributes processor you’ve created to the list of processors under pipelines. Place it after the batch processor and before the queued_retry processor, as the attributes/settenant processor is placed in the following code sample:

    service:
      pipelines:
        traces:
          receivers: ...
          processors: [...,  batch, attributes/settenant,  queued_retry, ...]
          ...
    

Understand host-specific span tags 🔗

If you send trace data to the Splunk Distribution of OpenTelemetry Collector, the collector automatically adds a host span tag to every span. This helps you identify which infrastructure component each span uses. The host span tag allows the application to render key infrastructure metrics and link to default dashboards for infrastructure components. This can help you monitor the performance of your applications at the infrastructure level. The host span tag value is the hostname or unique resource identifier for the infrastructure component.

In addition to the host span tag, the OpenTelemetry Collector automatically adds certain span tags to every span captured on that host, according to the type of host. These additional span tags provide more information about infrastructure components each span uses, render corresponding infrastructure metrics, and link to more complete dashboards for the underlying infrastructure component.

The following table provides examples of host-specific span tags:

Span tag

Description

AWSUniqueId or gcp_id

Unique resource identifier for cloud providers.

container_id

Unique resource identifier for Docker containers.

kubernetes_pod_uid

Unique resource identifier for a resource in a Kubernetes cluster.