Docs » Instrument serverless functions for Splunk Observability Cloud » Instrument AWS Lambda functions for Splunk Observability Cloud » Troubleshoot the Splunk OpenTelemetry Lambda layer

Troubleshoot the Splunk OpenTelemetry Lambda layer 🔗

If your instrumented AWS Lambda function is not sending data to Splunk Observability Cloud, or data is missing, follow these steps to identify and resolve the issue.

No data appears in Splunk Observability Cloud 🔗

If no data from your instrumented AWS Lambda function appears in Splunk Observability Cloud, try the following steps:

  1. Check the CloudWatch metrics of your AWS Lambda Function. Make sure the Lambda function is responding to invocations. You can also check for errors.

  2. Make sure that you specified the SPLUNK_REALM and SPLUNK_ACCESS_TOKEN environment variables. See Configure the Splunk OpenTelemetry Lambda layer.

  3. Try increasing the value of the OTEL_INSTRUMENTATION_AWS_LAMBDA_FLUSH_TIMEOUT environment variable if the back end or network is slow.

  4. Activate verbose logging of the extension. See Activate debug logging.

No data points or metrics in Splunk Observability Cloud 🔗

If no metric data from your instrumented AWS Lambda function appears in Splunk Observability Cloud, try the following steps:

  1. Check the CloudWatch metrics of your AWS Lambda Function. Make sure the Lambda function is responding to invocations. You can also check for errors.

  2. Make sure that you specified the SPLUNK_REALM and SPLUNK_ACCESS_TOKEN environment variables. See Configure the Splunk OpenTelemetry Lambda layer.

  3. The extension might be sending data points with significant delay due to buffering mode. See Metrics configuration.

  4. Activate verbose logging of the extension. See Activate debug logging.

Error about SPLUNK_ACCESS_TOKEN and SPLUNK_REALM 🔗

If the following error appears, you must set the value of the SPLUNK_ACCESS_TOKEN environment variable:

[ERROR] SPLUNK_REALM is set, but SPLUNK_ACCESS_TOKEN is not set. To export data to Splunk Observability Cloud, define a Splunk Access Token.

For more information, see General settings.

Error about exporter endpoint and SPLUNK_REALM 🔗

If the following error appears, you must either set the value of the SPLUNK_REALM and SPLUNK_ACCESS_TOKEN environment variables, or define an exporter endpoint:

[ERROR] Exporter endpoint must be set when SPLUNK_REALM is not set. To export data, either set a realm and access token or a custom exporter endpoint.

For instructions on how to define a custom exporter endpoint, see Exporters configuration.

Deactivate instrumentations that load automatically 🔗

Some of the wrappers included in the Splunk OpenTelemetry Lambda layer load instrumentations for popular libraries or frameworks automatically. To deactivate instrumentations that load automatically, follow these steps:

Enter the instrumentations you want to deactivate as comma-separated values for the OTEL_PYTHON_DISABLED_INSTRUMENTATIONS environment variable. For a list of automatically loaded instrumentations, see the requirements list in the OpenTelemetry repository on GitHub: https://github.com/open-telemetry/opentelemetry-lambda/blob/main/python/src/otel/otel_sdk/nodeps-requirements.txt.

Activate debug logging 🔗

If trace data for your function still doesn’t appear in Splunk Observability Cloud, activate logging to collect debugging information:

  1. Set the OTEL_LAMBDA_LOG_LEVEL environment variable to DEBUG for your instrumented function.

  2. Check AWS CloudWatch for spans.

  3. Search for specific spans in the back end.

For metric data, follow these steps:

  1. Set the VERBOSE environment variable to true.

  2. Set the HTTP_TRACING environment variable to true.

  3. Search for relevant log messages in AWS CloudWatch.

Serverless Framework support 🔗

Some features of the Serverless Framework might impact OpenTelemetry tracing of Python Lambda functions.

Python libraries compression 🔗

The zip feature of pythonRequirements allows packing and deploying Lambda dependencies as compressed files. To instrument packages compressed using the Serverless Framework, set the SPLUNK_LAMBDA_SLS_ZIP environment variable to true. For more information, see https://github.com/serverless/serverless-python-requirements#dealing-with-lambdas-size-limitations on GitHub.

Slim feature 🔗

The Slim feature reduces the size of Lambda packages by removing some files, including dist-info folders. Some of the files removed by the Slim feature are required by the OpenTelemetry Python autoinstrumentation. Deactivate the slim option in your serverless.yml file or define custom slimPatterns. For more information, see https://github.com/serverless/serverless-python-requirements#slim-package on GitHub.

This page was last updated on Jun 07, 2024.