Docs » Supported integrations in Splunk Observability Cloud » Instrument back-end applications to send spans to Splunk APM » Instrument Python applications for Splunk Observability Cloud » Troubleshoot Python instrumentation for Splunk Observability Cloud

Troubleshoot Python instrumentation for Splunk Observability Cloud 🔗

When you instrument a Python application using the Splunk Distribution of OpenTelemetry Python and you don’t see your data in Splunk Observability Cloud, follow these troubleshooting steps.

Steps for troubleshooting Python OpenTelemetry issues 🔗

The following steps can help you troubleshoot Python agent issues:

  1. Check that your pip install directory is in PATH

  2. Activate debug logging

Check that your pip install directory is in PATH 🔗

To run splunk-py-trace and splunk-py-trace-bootstrap, the install directory of pip must be on your system’s PATH environment variable.

If pip installs packages into your user local environment, add the user base bin directory to PATH.

  1. Find out your Python user base by running python -m site --user-base.

  2. Add the user base directory to your path:

    export PATH="<user-base-path>:$PATH"

Activate debug logging 🔗

Activating debug logging can help you troubleshoot Python instrumentation issues.

To activate logging, import the logging module and configure the logging level to DEBUG:

import logging

logging.basicConfig(level=logging.DEBUG)

When you run the agent with debug logging activated, debug information is sent to the console (stderr). Debug log entries look like the following example:

...
[opentelemetry.auto.trace 2021-10-10 10:57:05:814 +0200] [main] DEBUG io.opencensus.tags.Tags - <Could not load lite implementation for TagsComponent, now using default implementation for TagsComponent.3>
[opentelemetry.auto.trace 2021-10-10 10:57:05:722 +0200] [main] DEBUG io.grpc.netty.shaded.io.netty.util.internal.PlatformDependent0 - direct buffer constructor: unavailable
...

While not all debug entries might be relevant to the issue affecting your Python instrumentation, the root cause is likely to appear in your debug log.

Note

Activate debug logging only when needed. Debug mode requires more resources.

Trace exporter issues 🔗

By default, the Splunk Distribution of OpenTelemetry Python uses the OTLP exporter. Any issue affecting the export of traces produces an error in the debug logs.

OTLP can’t export spans 🔗

The following error in the logs means that the agent can’t send trace data to the OpenTelemetry Collector:

DEBUG:opentelemetry.exporter.otlp.proto.grpc.exporter:Waiting 1s before retrying export of span
DEBUG:opentelemetry.exporter.otlp.proto.grpc.exporter:Waiting 2s before retrying export of span

To troubleshoot the lack of connectivity between the OTLP exporter and the OTel Collector, try the following:

  1. Make sure that OTEL_EXPORTER_OTLP_ENDPOINT points to the correct OpenTelemetry Collector instance host.

  2. Check that your OTel Collector instance is configured and running. See Troubleshoot the Splunk OpenTelemetry Collector.

  3. Check that the OTLP gRPC receiver is activated in the OTel Collector and plugged into the traces pipeline.

  4. Check that the OTel Collector points to the following address: http://<host>:4317. Verify that your URL is correct.

Channel pipeline error 🔗

If you’re seeing the following error in your logs, it might mean that the Python agent is trying to send trace data to the Splunk Observability Cloud ingest endpoint, which is not yet supported by OTLP:

E0908 16:23:32.337704280    5881 ssl_transport_security.cc:1468] Handshake failed with fatal error SSL_ERROR_SSL: error:10000095:SSL routines:OPENSSL_internal:ERROR_PARSING_EXTENSION.
E0908 16:23:32.556405854    5881 ssl_transport_security.cc:1468] Handshake failed with fatal error SSL_ERROR_SSL: error:10000095:SSL routines:OPENSSL_internal:ERROR_PARSING_EXTENSION.

To solve this issue, use another exporter or send data to Splunk Observability Cloud directly. See Exporters configuration and Send data directly to Splunk Observability Cloud.

Jaeger can’t export spans 🔗

Note

The Jaeger exporter is no longer supported for Python. Use the OTLP exporter or send data to Splunk Observability Cloud directly. See Send data directly to Splunk Observability Cloud for more information.

If you’re exporting trace data using the Jaeger exporter, errors in your logs might mean the Python agent can’t send trace data to the OTel Collector, Smart Agent (now deprecated), or Splunk Cloud Platform:

To troubleshoot the lack of connectivity between Jaeger and Splunk Observability Cloud, try the following:

  1. Make sure that OTEL_EXPORTER_JAEGER_ENDPOINT points to a OpenTelemetry Collector or your Collector or Smart Agent (deprecated) configuration instance, or to the Splunk Ingest URL. See the Splunk Ingest URL summary in Summary of Splunk Observability Cloud API Endpoints .

  2. Check that the OTel Collector or Smart Agent instance is configured and running.

  3. Check that the Jaeger Thrift HTTP receiver is activated and plugged into the traces pipeline. See Exposed ports and endpoints.

  4. Check that the endpoint is correct. The OpenTelemetry Collector and Smart Agent use different ports and paths by default. For the Jaeger receiver, the OTel Collector uses http://<host>:14268/api/traces, while the Smart Agent uses http://<host>:9080/v1/trace.

401 error when sending spans 🔗

If you send traces directly to Splunk Observability Cloud and receive a 401 error code, the authentication token specified in SPLUNK_ACCESS_TOKEN is invalid. The following are possible reasons:

  • The value is null.

  • The value is not a well-formed token.

  • The token is not an access token that has authScope set to ingest.

Make sure that you’re using a valid Splunk access token when sending data directly to your Splunk platform instance. See Retrieve and manage user API access tokens using Splunk Observability Cloud.

If you are a Splunk Observability Cloud customer and are not able to see your data in Splunk Observability Cloud, you can get help in the following ways.

Available to Splunk Observability Cloud customers

Available to prospective customers and free trial users

  • Ask a question and get answers through community support at Splunk Answers .

  • Join the Splunk #observability user group Slack channel to communicate with customers, partners, and Splunk employees worldwide. To join, see Chat groups in the Get Started with Splunk Community manual.

To learn about even more support options, see Splunk Customer Success .