Docs » Manage services, spans, and traces in Splunk APM » Analyze the performance of inferred services

Analyze the performance of inferred services 🔗

In certain cases, a remote service might not have tracing turned on, either because it is not instrumented yet, or because instrumentation is not possible. Splunk APM can infer the presence of the remote service, or inferred service, if the span calling the remote service has the necessary information.

Splunk APM adds an inferred span to the relevant trace to represent an operation occurring in an inferred service. For further details on how Splunk APM infers services, see How does Splunk APM identify inferred services?.

The following table describes common types of inferred services:

Type

Example

Notes
Generic service
(dotted circle in the service map)

AWS service
This service type includes external service providers, generic remote procedure calls (RPCs), and HTTP services. See the following sections to learn more about each of these types:
If your system interacts with a large number of HTTP services, inferring HTTP services might increase the cardinality of your Troubleshooting MetricSets (TMS). If so, you might turn off inferring HTTP services in your org to prevent excessive cardinality. See Manage inferring HTTP services to manage inferred HTTP services.
Publisher/subscriber queues (pub/sub)
(dotted arrow in the service map)

Kafka, Kinesis

To learn more about inferred pub/sub queues, see Inferred publisher/subscriber (pub/sub) queues.
Database
(dotted cylinder in the service map)

MySQL, AWS S3, Redis, MongoDB

In addition to the details described in What information about inferred services is available?, Splunk APM provides additional analytics for inferred SQL databases. See Monitor Database Query Performance to learn more about database insights in Splunk APM.

What information about inferred services is available? 🔗

Because inferred services are not instrumented, Splunk APM only has access to spans from instrumented services that call inferred services. Splunk APM places inferred services in their inferred locations in the service map and provides Troubleshooting MetricSets (TMS), or request, error, and duration (RED) metrics, based on the associated client-side spans. You can continue to troubleshoot with these metrics in Tag Spotlight. If the inferred service is servicing other uninstrumented services, these reported RED metrics might not provide a complete picture of the inferred service’s interactions.

Splunk APM does not create Monitoring MetricSets (MMS) for inferred services because MMS are created in real time from spans where span.kind = SERVER or span.kind = CONSUMER, and Splunk APM does not receive server or consumer-side spans from uninstrumented services. This means that even though inferred services appear in the list of services on the landing page, you can’t get real-time MMS data on inferred services in the APM landing page or in dashboards and alerts, and you can’t create detectors based on inferred services.

Viewing inferred services in the service map 🔗

Inferred services, pub/subs, and databases appear in their inferred locations in the service map. When you select on an inferred service in the service map, you can view its service type, service names, and request, error rate, and latency (RED) metrics. Metrics are based on referring spans from instrumented services and might not provide a complete picture of the inferred service’s interactions.

Viewing inferred services in trace view 🔗

Inferred services, pub/subs, and databases also appear as spans in the Trace Waterfall view. They appear in a gray box with italicized print, as in the following screenshot:

This screenshot shows an example of inferred spans appearing in Trace View.

When you select an inferred span in the Trace Waterfall, it expands to show the metadata of the corresponding parent span. The length of the operation represented as the gray striped bar in the waterfall visualization is also inherited from the parent span and might not be exactly representative of the operation duration in the inferred service.

How does Splunk APM identify inferred services? 🔗

When a client or producer span doesn’t have a corresponding server span, Splunk APM checks whether the unpaired span contains tags that indicate interaction with an uninstrumented service.

To identify an inferred service, Splunk APM first checks for tags that indicate the type of the inferred service, and then checks for tags that indicate the service name. After Splunk APM identifies a client or producer span with a tag or tags that indicate interaction with one of these service types, it creates an inferred span to represent the operation in the uninstrumented service.

The table in Types of inferred services and how they’re inferred provides a list of types of inferred services and the tags and the inference methods that Splunk APM uses to identify each type of inferred service. In the case of inferred pub/sub services, the inferred span inherits the metadata from the corresponding client or producer span and is attached directly to that span.

For inferred service types other than pub/sub, Splunk APM applies additional logic to ensure it captures only the most important application-level information. If there are multiple client spans without a corresponding server span, the inferred span inherits the metadata from the parent-most of these client spans. The inferred service span is then attached to the most recent of these client spans. Otherwise, if there is a single client span without a corresponding server span, the inferred span inherits the metadata from that client span and is attached to that same span.

Manage inferring HTTP services 🔗

You can configure Inferring HTTP services to control over inferred services appear in the service map and generate Troubleshooting MetricSets.

The advantage of inferring HTTP services is that you have visibility into their interactions with instrumented services. The potential downside is that if there are a lot of different HTTP services called from your system, they might crowd the service map and increase TMS cardinality.

Splunk APM administrators can manage inferred HTTP services in the APM Configuration, providing flexibility based on your particular system.

Prerequisite 🔗

To configure inferred HTTP services, you must have an admin role.

Configure Splunk APM to infer HTTP services 🔗

Follow these steps to configure inferred HTTP services in Splunk APM:

  1. From the Splunk APM Landing page, select APM Configuration and select APM Service & Traces from the menu. The APM Services & Traces page opens.

  2. Under Inferred services, find the row for Inferred HTTP service and select Configure. The configuration dialog box opens.

  3. In the Configure Inferred HTTP Services configuration dialog box, select Infer HTTP services.

  4. Save your changes.

Note

Inferring HTTP services generates Troubleshooting MetricSets for each new inferred service and therefore increases Troubleshooting MetricSet cardinality. Select Subscription Usage to view your current cardinality usage. See Troubleshoot cardinality in Monitoring MetricSets to learn more about managing cardinality.

Types of inferred services and how they’re inferred 🔗

The following table provides a list of types of inferred services and the tags and the inference methods that Splunk APM uses to identify each type of inferred service. The span tags used in these inference rules are based on OpenTelemetry semantic conventions.

Splunk APM infers the following types of services:

Inferred HTTP services 🔗

When Splunk APM infers an HTTP service, it means an instrumented service is talking to a remote HTTP endpoint.

To assign a service name for an inferred HTTP service, Splunk APM does the following:

  1. Verify that the span.kind of the referring span is equal to CLIENT.

  2. If one or more of http.host, http.url or net.peer.name exist, and peer.service exists as well, use peer.service for the service name. This ensures that the peer.service is an HTTP service.

  3. Look for the service name in the following tags, in this order:

    1. http.host: host name extracted as-is

    2. peer.hostname: host name extracted as-is

    3. peer.address: host name is extracted from the URL

    4. http.url: host name is extracted from the URL

    5. net.peer.name: host name extracted as-is

  4. If any of these tags are found, infer the service name from the first appearing tag. If none of these tags are found, the span is not considered related to an inferred HTTP service.

Note

To reduce noise in the service map and managing cardinality, Splunk APM excludes services without a host name or that use their IP address as host name. If you need to turn on IP addresses, contact your sales representative.

Inferred RPC services 🔗

When Splunk APM infers an RPC service, it means an instrumented service is making a remote procedure call.

To infer an RPC service, Splunk APM does the following:

  1. Verify that the span.kind of the referring span is equal to CLIENT.

  2. Verify that the referring span contains the rpc.system span tag. This tag is used to identify the remote system, such as grpc, java_rmi, or wcf.

  3. Look for the service name in the rpc.service, net.peer.name and rpc.system` tags, in that order.

  4. If any of these tags are found, infer the service name from the first appearing tag. If none of these tags are found, the span is not considered related to an inferred RPC service.

Generic inferred services 🔗

This is a catch-all layer to infer generic services using the peer.service span tag.

To infer a generic service from a client span, Splunk APM does the following:

  1. Verify that the span.kind of the referring span is equal to CLIENT.

  2. Look for the service name in the peer.service tag

  3. If the peer.service tag exists, infer the service name from it. If the peer.service tag doesn’t exist, the span isn’t considered to be related to a generic inferred service.

Note on AWS services: To identify AWS services, the span must contain http.url. Splunk APM applies heuristics on this tag’s value to determine the AWS Service type from the URL.

Inferred publisher/subscriber (pub/sub) queues 🔗

When Splunk APM infers a publisher/subscriber queue, it means an instrumented service is interacting with an uninstrumented pub/sub. To identify an inferred pub/sub, Splunk APM does the following:

  1. Verifies that the span.kind of the referring span is equal to PRODUCER or CLIENT.

  2. Verifies that the span contains either messaging.destination (in libraries that support OpenTelemetry semantic conventions version 1.16.0 or lower) or messaging.destination.name (in libraries that support OpenTelemetry semantic conventions version 1.17.0 or higher). The value of these tags is used to specify the name of the topic or channel that messages are sent to.

    1. If both messaging.system and messaging.destination.name exist, the inferred service name is equal to <Value of messaging.system tag>:<Value of messaging.destination.name tag>.

    2. If messaging.system is null, the inferred service name is equal to <Value of messaging.destination.name tag>.

    3. If messaging.destination.name is null, the inferred service name is equal to <Value of messaging.system tag>.

Inferred databases 🔗

When Splunk APM infers a database, it means an instrumented service is making a call to an uninstrumented database.

To identify a database, the kind of the referring span must be equal to client, and the span must contain at least one of the following tags:

  • db.system

  • db.name

  • db.type

  • db.instance

To determine the name of an inferred database, Splunk APM applies this logic in the following order:

  1. If the db.system tag exists, its value is used to specify the type of database being queried, for example mysql, redis, and so on. If only this tag is present, its value is also used as the service.name for the inferred database.

  2. If the db.name tag exists, its value is concatenated with db.system to form the name of the inferred service: db.system:db.name (for example mysql:sql_db_1).

  3. If the db.connection_string tag is present and its value conforms to a known format such as Java database connectivity (JDBC), Splunk APM extracts the database name portion of the url and concatenates it with the value of db.system to form the database name, such as mysql:dbname. If the value of db.connection_string does not conform to a known format or the database portion cannot be extracted and db.name also does not exist, Splunk APM uses the raw value of db.connection_string as the database name. If db.system also exists, the two values are concatenated.

Splunk APM also provides additional analytics for supported SQL databases. See Monitor Database Query Performance to learn more.