Docs » Manage services, spans, and traces in Splunk APM

Manage services, spans, and traces in Splunk APM 🔗

Spans and traces form the backbone of application monitoring in Splunk APM. Use the following topics to learn about spans and traces and manage your spans and traces effectively, so that you can get the most out of APM full-fidelity tracing:

What are services?¶

Services are the key components of the systems you can monitor with Splunk APM.

What are traces and spans?¶

A trace is a collection of operations that represents a unique transaction handled by an application and its constituent services. A span represents a single operation within a trace.

Splunk APM collects incoming spans into traces and analyzes them to give you full fidelity access to your application data. After all spans of a trace are ingested and analyzed, the trace is available to view in all parts of APM.

The following image illustrates the relationship between traces and spans:

This image shows a trace represented by a series of multicolored bars labeled with the letters A, B, C, D, and E. Each lettered bar represents a single span. The spans are organized to visually represent a hierarchical relationship in which span A is the parent span and the subsequent spans are its children.

A span might refer to another span as its parent, indicating a relationship between operations involved in the trace. In the preceding image, span A is a parent span, and span B is a child span. This relationship might indicate that, for example, span A makes a service call that triggers the operation captured by span B. In this image, span C is also a child of span B, and so on.

Span metadata¶

Each span contains metadata about the operation captured by the span and the service in which the operation took place.

Span metadata includes a set of basic metadata including information such as the service and operation. Spans also include span tags, which provide additional operation-specific metadata. See the Span tags section in this topic to learn more.

Each span contains the following basic metadata:

Span tag key

Example value

Description

Service

checkoutservice

Name of the service associated with the span

Operation

cart/checkout

Name of the operation that the span represents

Span ID

976gh68e482k8s0m

Unique 16-character identifier for the specific span

Parent ID (if applicable)

756e643so999cl4

Unique 16-character identifier for the span’s parent span, if it has one

Time (Start - End)

2022-01-05T11:51:56.360229 - 2022-01-05T11:51:56.932772

Starting and ending timestamps for the span

Duration

573ms

Duration of the span (in seconds, milliseconds, or microseconds)

Relative Start

+972µs

Delay between the start of the parent trace and the start of this particular span

Span tags¶

Span tags are key-value pairs that provide additional information and context about the operations a span represents. Both the keys and values are strings, and span tag keys for a single span must be unique. You can use span tags to query and filter traces, or to get information about the spans of a trace during troubleshooting.

You can add custom span tags through the OpenTelemetry Collector, or when you instrument an application. For more information about using span tags to analyze service performance, see Analyze services with span tags and MetricSets in Splunk APM.

Span tags are most useful when they follow a simple, dependable system of naming conventions. See Follow span tag naming conventions to learn about OpenTelemetry naming conventions for span tags.

Note

Span tags in Splunk APM are distinct from metadata tags in Splunk Infrastructure Monitoring, which are searchable labels or keywords you can assign to metric dimensions in the form of strings rather than as key-value pairs. To learn more about metadata tags, see Infrastructure Monitoring tags.

This page was last updated on Nov 08, 2024.