Add context to spans with span tags in Splunk APM 🔗

Enrich the context of the spans you send to Splunk APM by adding span tags. Span tags are key-value pairs that provide additional metadata about spans in a trace. In OpenTelemetry, span tags are attributes.

There are two ways to add span tags to your spans:

Instrument your application to create span tags. This option gives you the most flexibility at the per-application level. See Instrument your application code to add tags to spans to learn how.

Add span tags as OpenTelemetry attributes to spans when you send data to the Splunk Distribution of OpenTelemetry Collector. This option lets you add span tags to spans in bulk. See Add span tags with the Splunk Distribution of OpenTelemetry Collector to learn how.

If you deploy the Splunk Distribution of OpenTelemetry Collector as a gateway to centrally manage data collection from multiple services, you might want to instrument your application to define span tags and manage other span tags with the Collector.

Prerequisite 🔗

To add span tags, you must have an admin role.

Follow span tag naming conventions 🔗

Tags provide more value when you use a naming convention. Define clear tag names to use for all your applications. As a starting point, OpenTelemetry provides a set of semantic conventions you can use with your spans and traces. For more information, see OpenTelemetry Trace Semantic Conventions on GitHub.

Add tags to spans 🔗

The following sections describe two ways to add tags to your spans:

Instrument your application code to add tags to spans
Add span tags with the Splunk Distribution of OpenTelemetry Collector

Begin by considering where to add your span tags.

Note

Deciding where to add a span tag

When you add span tags to spans via application instrumentation, you have the most control at the per-application level. The tradeoff is that it can be more time-consuming to add an attribute via instrumentation for each of your applications.

Adding span tags in a downstream OpenTelemetry Collector can save time, at the cost of granularity. If a tag is applicable to 100% of the data received by the OpenTelemetry Collector in which you are adding it, adding the tag in your OpenTelemetry Collector config file is fastest. But if you need to apply logic to differentiate the spans that receive a tag via the OpenTelemetry Collector, it is likely faster to add the tags at the application level via instrumentation.

For instance, if multiple applications exist on the same host in a K8s deployment, but in different environments (for instance, production, development, and staging), setting the deployment.environment tag using the instrumentation library allows you to differentiate environments among spans from the same host.

The deployment.environment span tag is particularly useful, as it lets you filter your entire APM experience by deployment environment. To learn more about environments in Splunk APM, see Set up deployment environments in Splunk APM.

Instrument your application code to add tags to spans 🔗

When you add span tags, you can specify tags on a per-application basis. How you instrument code to create span tags or attributes depends on the programming language and instrumentation library you’re using.

The following examples show how to create a custom tag for an existing span:

// Splunk Distribution of OpenTelemetry Java

import io.opentelemetry.api.trace.Span;

Span customizedSpan = Span.current();

customizedSpan.setAttribute("my.attribute","value");

// You can also set global tags using the OTEL_RESOURCE_ATTRIBUTES
// environment variable, which accepts a list of comma-separated key-value
// pairs. For example, key1:val1,key2:val2.

# Splunk Distribution of OpenTelemetry Python

from opentelemetry import trace

customizedSpan = trace.get_current_span()

customizedSpan.set_attribute("my.attribute", "value")

# You can also set global tags using the OTEL_RESOURCE_ATTRIBUTES
# environment variable, which accepts a list of comma-separated key-value
# pairs. For example, key1:val1,key2:val2.

// Splunk Distribution of OpenTelemetry JS

const { context, trace } = require('@opentelemetry/api');

// A span must already exist in the context

const customizedSpan = trace.getSpan(context.active());

customizedSpan.setAttribute('my.attribute', 'value');

// You can also set global tags using the OTEL_RESOURCE_ATTRIBUTES
// environment variable, which accepts a list of comma-separated key-value
// pairs. For example, key1:val1,key2:val2.

// Splunk Distribution of OpenTelemetry .NET

using var myActivity = MyActivitySource.StartActivity("SayHello");

activity?.SetTag("operation.value", 1);
activity?.SetTag("operation.name", "Saying hello!");
activity?.SetTag("operation.other-stuff", new int[] { 1, 2, 3 });

// You can also set global tags using the OTEL_RESOURCE_ATTRIBUTES
// environment variable, which accepts a list of comma-separated key-value
// pairs. For example, key1:val1,key2:val2.

import (
   // ...
   "go.opentelemetry.io/otel"
)

func myFunc(ctx context.Context) {

   // Create a named tracer
   tracer := otel.Tracer("example.com/myFunc")

   // Create a span with custom attributes
   ctx, span = tracer.Start(ctx, "attributesAtCreation", trace.WithAttributes(attribute.String("hello", "splunk")))
   defer span.End()

   // Add attributes after creation
   span.SetAttributes(attribute.Bool("isTrue", true), attribute.String("stringAttr", "Hello there!"))

   // Other activities
}

// You can also set global tags using the OTEL_RESOURCE_ATTRIBUTES
// environment variable, which accepts a list of comma-separated key-value
// pairs. For example, key1:val1,key2:val2.

# OpenTelemetry Ruby

require "opentelemetry/sdk"

current_span = OpenTelemetry::Trace.current_span

current_span.set_attribute("animals", ["elephant", "tiger"])

# You can also set global tags using the OTEL_RESOURCE_ATTRIBUTES
# environment variable, which accepts a list of comma-separated key-value
# pairs. For example, key1:val1,key2:val2.

<?php

// OpenTelemetry PHP

use SignalFx\GlobalTracer;

private function rollOnce() {
   $parent = OpenTelemetry\API\Trace\Span::getCurrent();
   $scope = $parent->activate();
   try {
      $span = $this->tracer->spanBuilder("rollTheDice")->startSpan();
      $result = random_int(1, 6);
      $span->setAttribute('dicelib.rolled', $result);
      $span->end();
   } finally {
      $scope->detach();
   }
   return $result;
}

// You can also set global tags using the SIGNALFX_TRACE_GLOBAL_TAGS
// environment variable, which accepts a list of comma-separated key-value
// pairs. For example: key1:val1,key2:val2.
?>

Add span tags with the Splunk Distribution of OpenTelemetry Collector 🔗

To add a span tag to spans received by the Splunk Distribution of OpenTelemetry Collector, you can use the attributes processor in your OpenTelemetry Collector configuration YAML file.

The generic attributes processor is called attributes. Any subsequent attributes/<NAME> processors are named instances of the attributes processor. See Attributes processor for more information.

Follow these steps to define a new attributes processor and add it to your pipeline:

Define an attributes processor that adds your desired span tag. There are two ways to do this:
Use the insert action to set a new key-value pair.

For instance, the following code sample adds these key-value pair enduser.role:"admin" in spans where the key enduser.role doesn’t already exist:
processors:
  ...
  attributes/setenduser.role:
    actions:
    - key: enduser.role
        value: "admin"
        action: insert
Use the upsert action to copy a value from an existing key in the spans and add it to a new key, overriding any existing values for that key.

For instance, the following code sample copies the value from the existing myTenant key to the tenant key and overrides any existing values for the tenant key:
processors:
  ...
  attributes/settenant:
    actions:
    - key: tenant
        from_attribute: myTenant
        action: upsert
Add the attributes processor you’ve created to the list of processors under pipelines. Place it after the batch processor and before the queued_retry processor, as the attributes/settenant processor is placed in the following code sample:
service:
  pipelines:
    traces:
      receivers: ...
      processors: [...,  batch, attributes/settenant,  queued_retry, ...]
      ...

Where do host-specific span tags come from? 🔗

The Splunk Distribution of OpenTelemetry Collector automatically adds a host span tag to every span to identify which infrastructure component each span uses. The host span tag value is often the hostname or unique resource identifier for the infrastructure component.

The host span tag allows Splunk APM to render key infrastructure metrics and link to default dashboards for infrastructure components. This can help you more easily monitor the performance of your applications at the infrastructure level and use interactions between Splunk APM and Splunk Infrastructure Monitoring.

In addition to the host span tag, the OpenTelemetry Collector automatically adds certain span tags to every span captured on that host, according to the type of host. These additional span tags provide more information about infrastructure components each span uses, render corresponding infrastructure metrics, and link to more complete dashboards for the underlying infrastructure component.

The following table provides examples of host-specific span tags:

Span tag	Description
`AWSUniqueId` or `gcp_id`	Unique resource identifier for cloud providers.
`container_id`	Unique resource identifier for Docker containers.
`kubernetes_pod_uid`	Unique resource identifier for a resource in a Kubernetes cluster.

Related Topics