Docs » Manage services, spans, and traces in Splunk APM » Analyze the performance of inferred services

Analyze the performance of inferred services 🔗

In certain cases, a remote service might not be sending spans directly to Splunk APM, either because it is not instrumented yet, or because instrumentation is not possible. In these cases, Splunk APM can infer the presence of the remote service, or inferred service, if the span calling this remote service has the necessary information.

Splunk APM adds an inferred span to the relevant trace to represent an operation occurring in an inferred service. For further details on how services are inferred, see How does Splunk APM identify inferred services? below.

The following table describes common types of inferred services:

Type

Example

Notes

Generic service
(dotted circle in the service map)

AWS service

This service type includes external service providers, generic remote procedure calls (RPCs), and HTTP services. See the following sections to learn more about each of these types:
If your system interacts with a large number of HTTP services, inferring HTTP services might increase TMS cardinality, and inferring HTTP services might be disabled in your org to prevent excessive cardinality. See Manage inferring HTTP services in this topic to manage inferred HTTP services.
Publisher/subscriber queues (pub/subs)
(dotted arrow in the service map)

Kafka, Kinesis

To learn more about inferred pub/sub queues, see Inferred publisher/subscriber (pub/sub) queues.

Database
(dotted cylinder in the service map)

mySQL, AWS S3, Redis, MongoDB

In addition to the details described in What information about inferred services is available?, Splunk APM provides additional analytics for inferred SQL databases. See Monitor Database Query Performance to learn more about database insights in Splunk APM.

What information about inferred services is available? 🔗

Because inferred services are not instrumented, Splunk APM only has access to spans from instrumented services that call inferred services. Splunk APM places inferred services in their inferred locations in the service map and provides Troubleshooting MetricSets (TMS), or request, error, and duration (RED) metrics, based on the associated client-side spans. You can continue to troubleshoot with these metrics in Tag Spotlight. If the inferred service is servicing other uninstrumented services, these reported RED metrics might not provide a complete picture of the inferred service’s interactions.

Splunk APM does not create Monitoring MetricSets (MMS) for inferred services because MMS are created in real time from spans where span.kind = SERVER or span.kind = CONSUMER, and Splunk APM does not receive server- or consumer-side spans from uninstrumented services. This means that even though inferred services appear in the list of services on the landing page, you can’t get real-time MMS data on inferred services in the APM landing page or in dashboards and alerts, and you can’t create detectors based on inferred services.

Viewing inferred services in the service map 🔗

Inferred services, pub/subs, and databases appear in their inferred locations in the service map. When you select on an inferred service in the service map, you can view its service type, service name(s), and request, error rate, and latency (RED) metrics. Note that these metrics are based on referring spans from instrumented services and may not provide a complete picture of the inferred service’s interactions.

Viewing inferred services in trace view 🔗

Inferred services, pub/subs, and databases also appear as spans in the Trace Waterfall view. They are shown in a gray box with italicized print, as in the following screenshot.

This screenshot shows an example of inferred spans appearing in Trace View.

When you select an inferred span in the Trace Waterfall, it expands to show the metadata of the corresponding parent span. The length of the operation represented as the gray striped bar in the waterfall visualization is also inherited from the parent span and might not be exactly representative of the operation duration in the inferred service.

How does Splunk APM identify inferred services? 🔗

When a client or producer span does not have a corresponding server span, Splunk APM checks whether the unpaired span contains tags that indicate interaction with an uninstrumented service.

To identify an inferred service, Splunk APM first checks for tags that indicate the type of the inferred service, and then checks for tags that indicate the service name. Once Splunk APM identifies a client or producer span with a tag or tags that indicate interaction with one of these service types, it creates an inferred span to represent the operation in the uninstrumented service.

The table in Types of inferred services and how they’re inferred below provides a list of types of inferred services and the tags and the inference methods that Splunk APM uses to identify each type of inferred service.

In the case of inferred pub/sub services, the inferred span inherits the metadata from the corresponding client or producer span and is attached directly to that span.

For inferred service types other than pub/sub, Splunk APM applies additional logic to ensure it captures only the most important application-level information. If there are multiple client spans without a corresponding server span, the inferred span inherits the metadata from the parent-most of these client spans. The inferred service span is then attached to the most recent of these client spans. Otherwise, if there is just a single client span without a corresponding server span, the inferred span inherits the metadata from that client span and is attached to that same span.

Manage inferring HTTP services 🔗

Inferring HTTP services is configurable to give you control over whether inferred services appear in the service map and generate Troubleshooting MetricSets.

The advantage of inferring HTTP services is that you have visibility into their interactions with instrumented services. The potential downside is that if there are a lot of different HTTP services being called from your system, they may crowd the service map and increase TMS cardinality.

Splunk APM administrators can manage inferred HTTP services in the APM Configuration, providing flexibility based on your particular system.

Prerequisites 🔗

You need to be an administrator in Splunk APM to manage HTTP inferred services.

Configure Splunk APM to infer HTTP services 🔗

Follow these steps to configure inferred HTTP services in Splunk APM:

  1. From the Splunk APM Landing page, select APM Configuration and select APM Service & Traces from the drop-down menu. The APM Services & Traces page opens.

  2. Under Inferred services, find the row for Inferred HTTP service and select Configure. The configuration modal opens.

  3. In the Configure Inferred HTTP Services configuration modal, select the checkbox button for Infer HTTP services.

  4. Save your changes.

Note

Inferring HTTP services generates Troubleshooting MetricSets for each new inferred service and therefore increases Troubleshooting MetricSet cardinality. Visit the Usage Subscription page to view your current cardinality usage. See Troubleshoot cardinality in Monitoring MetricSets to learn more about managing cardinality.

Types of inferred services and how they’re inferred 🔗

The following table provides a list of types of inferred services and the tags and the inference methods that Splunk APM uses to identify each type of inferred service. The span tags used in these inference rules are based on OpenTelemetry semantic conventions.

Splunk APM infers the following types of services:

Inferred HTTP services 🔗

When Splunk APM infers an HTTP service, it means an instrumented service is talking to a remote HTTP endpoint.

To assign a service name for an inferred HTTP service, Splunk APM does the following:

  1. Verify that the span.kind of the referring span is equal to CLIENT.

  2. If one or more of http.host, http.url or net.peer.name exist, and peer.service exists as well, use peer.service for the service name. This ensures that the peer.service is really for an HTTP service.

  3. Look for the service name in the following tags, in this order:

    1. http.host: host name extracted as-is

    2. peer.hostname: host name extracted as-is

    3. peer.address: host name is extracted from the URL

    4. http.url: host name is extracted from the URL

    5. net.peer.name: host name extracted as-is

  4. If any of these tags are found, infer the service name from the first appearing tag. If none of these tags are found, the span is not considered related to an inferred HTTP service.

Inferred RPC services 🔗

When Splunk APM infers an RPC service, it means an instrumented service is making a remote procedure call.

To infer an RPC service, Splunk APM does the following:

  1. Verify that the span.kind of the referring span is equal to CLIENT.

  2. Verify that the referring span contains the rpc.system span tag. This tag is used to identify the remote system, such as grpc, java_rmi, or wcf.

  3. Look for the service name in the rpc.service, net.peer.name and rpc.system` tags, in that order.

  4. If any of these tags are found, infer the service name from the first appearing tag. If none of these tags are found, the span is not considered related to an inferred RPC service.

Generic inferred services 🔗

This is a catch-all layer to infer generic services using the peer.service span tag.

To infer a generic service from a client span, Splunk APM does the following:

  1. Verify that the span.kind of the referring span is equal to CLIENT.

  2. Look for the service name in the peer.service tag

  3. If the peer.service tag exists, infer the service name from it. If the peer.service tag does not exist, the span is not considered related to a generic inferred service.

Note on AWS services: To identify AWS services, the span must contain http.url. Splunk APM applies heuristics on this tag’s value to determine the AWS Service type from the URL.

Inferred publisher/subscriber (pub/sub) queues 🔗

When Splunk APM infers a publisher/subscriber queue, it means an instrumented service is interacting with an uninstrumented pub/sub.

The kind of the referring span is equal to producer or client.

To identify an inferred pub/sub, the span must contain messaging.destination. This tag’s value is used to specify the name of the topic or channel that messages are being sent to.

Inferred databases 🔗

When Splunk APM infers a database, it means an instrumented service is making a call to an uninstrumented database.

To identify a database, the kind of the referring span must be equal to client, and the span must contain at least one of the following tags:

  • db.system

  • db.name

  • db.type

  • db.instance

To determine the name of an inferred database, Splunk APM applies this logic in the following order:

  1. If the db.system tag exists, its value is used to specify the type of database being queried, e.g. mysql, redis, etc. If only this tag is present, its value is also used as the service.name for the inferred database.

  2. If the db.name tag exists, its value is concatenated with db.system to form the name of the inferred service: db.system:db.name (e.g. mysql:sql_db_1).

  3. If the db.connection_string tag is present and its value conforms to a known format such as Java database connectivity (JDBC), Splunk APM extracts the database name portion of the url and concatenates it with the value of db.system to form the database name, such as mysql:dbname. If the value of db.connection_string does not conform to a known format or the database portion cannot be extracted and db.name also does not exist, Splunk APM uses the raw value of db.connection_string as the database name. If db.system also exists, the two values are concatenated.

Splunk APM also provides additional analytics for supported SQL databases. See Monitor Database Query Performance to learn more.