Manage services, spans, and traces in Splunk APM 🔗
Spans and traces form the backbone of application monitoring in Splunk APM. Use the following topics to learn about spans and traces and manage your spans and traces effectively, so that you can get the most out of APM full-fidelity tracing:
Services are the key components of the systems you can monitor with Splunk APM.
A trace is a collection of operations that represents a unique transaction handled by an application and its constituent services. A span represents a single operation within a trace.
Splunk APM collects incoming spans into traces and analyzes them to give you full fidelity access to your application data. After all spans of a trace are ingested and analyzed, the trace is available to view in all parts of APM.
The following image illustrates the relationship between traces and spans:
A span might refer to another span as its parent, indicating a relationship between operations involved in the trace. In the preceding image, span A is a parent span, and span B is a child span. This relationship might indicate that, for example, span A makes a service call that triggers the operation captured by span B. In this image, span C is also a child of span B, and so on.
Each span contains metadata about the operation captured by the span and the service in which the operation took place.
Span metadata includes a set of basic metadata including information such as the service and operation. Spans also include span tags, which provide additional operation-specific metadata. See the Span tags section in this topic to learn more.
Each span contains the following basic metadata:
Span tag key |
Example value |
Description |
|---|---|---|
Service |
|
Name of the service associated with the span |
Operation |
|
Name of the operation that the span represents |
Span ID |
|
Unique 16-character identifier for the specific span |
Parent ID (if applicable) |
|
Unique 16-character identifier for the span’s parent span, if it has one |
Time (Start - End) |
|
Starting and ending timestamps for the span |
Duration |
|
Duration of the span (in seconds, milliseconds, or microseconds) |
Relative Start |
|
Delay between the start of the parent trace and the start of this particular span |
Span tags are key-value pairs that provide additional information and context about the operations a span represents. Both the keys and values are strings, and span tag keys for a single span must be unique. You can use span tags to query and filter traces, or to get information about the spans of a trace during troubleshooting.
You can add custom span tags through the OpenTelemetry Collector, or when you instrument an application. For more information about using span tags to analyze service performance, see Analyze services with span tags and MetricSets in Splunk APM.
Span tags are most useful when they follow a simple, dependable system of naming conventions. See Follow span tag naming conventions to learn about OpenTelemetry naming conventions for span tags.
Note
Note that span tags in Splunk APM are distinct from metadata tags in Splunk Infrastructure Monitoring, which are searchable labels or keywords you can assign to metric dimensions in the form of strings rather than as key-value pairs. To learn more about metadata tags, see Infrastructure Monitoring tags.
An identity represents a unique set of indexed span tags for a Splunk APM object, and always includes at least one service.
APM objects can generate multiple identities that correspond to the same APM object. If a set of indexed span tags for a span that corresponds to a certain APM object is unique, the APM object generates a new identity for the unique set of indexed span tags.
For example, a service myService reports a tenant span tag something for its endpoint /foo/bar, and doesn’t report a tenant span tag for its endpoint /another/endpoint. Because myService reports a tenant span tag for one endpoint and not another, it forces the endpoint without a specified tenant span tag to have a tenant span tag value of unknown. As a result, the service has two unique sets of span tags, and two identities.
An identity can represent any one of these APM objects:
APM object |
Example |
Description |
|---|---|---|
Service |
|
The name of a service you instrumented and are collecting traces from. |
Endpoint |
|
The first span for a service. |
Operation |
|
A span within a single service. |
Edge |
|
The span between two services. |
Workflow |
|
The endpoint where traces initiate. |