Docs » Instrument serverless functions for Splunk Observability Cloud » Instrument AWS Lambda functions for Splunk Observability Cloud » Instrument your AWS Lambda function for Splunk Observability Cloud

Instrument your AWS Lambda function for Splunk Observability Cloud ๐Ÿ”—

Use the Splunk OpenTelemetry Lambda layer to automatically instrument your AWS Lambda functions for many programming languages. To get started, use the guided setup or follow the instructions manually.

Generate customized instructions using the guided setup ๐Ÿ”—

To generate a template that instruments your Lambda function using the Splunk OpenTelemetry Lambda layer, use the AWS Lambda guided setup. To access the AWS Lambda guided setup, follow these steps:

  1. Log in to Splunk Observability Cloud.

  2. Open the AWS Lambda guided setup . Optionally, you can navigate to the guided setup on your own:

    1. In the navigation menu, select Data Management to open the Integrate Your Data page.

    2. In the integration filter menu, select All.

    3. In the Search field, search for AWS Lambda.

    4. Select the AWS Lambda tile to open the AWS Lambda guided setup.

Install the Splunk OpenTelemetry Lambda layer manually ๐Ÿ”—

Follow these instructions to install the Splunk OpenTelemetry Lambda layer:

To instrument .NET functions, see Instrument your .NET AWS Lambda function for Splunk Observability Cloud.

For Python functions using the Serverless Framework, see Serverless Framework support.

Check compatibility and requirements ๐Ÿ”—

The Splunk OpenTelemetry Lambda Layer supports the following runtimes in AWS Lambda:

  • Java 8.al2, 11, 17, and 21

  • Python 3.8 and 3.9

  • Node.js 14 and higher

  • Ruby 2.7

  • Go 1.20

The Lambda Layer requires 90 MB on-disk in standard x86_64 systems.

For more information, search for Lambda runtimes and other system requirements on the AWS documentation website.

Install the AWS Lambda layer ๐Ÿ”—

Follow these steps to instrument your function using the Splunk OpenTelemetry Lambda layer:

  1. In the AWS Lambda console, select the function that you want to instrument.

  2. In the Layers section, select Add a layer, then select Specify an ARN.

  3. Copy the Amazon Resource Name (ARN) for the region of your Lambda function from the list matching your architecture:

    <h3>Splunk OpenTelemetry Lambda Layer</h3>
    
    <pre>
    arn:aws:lambda:us-east-2:254067382080:layer:splunk-apm:573
    arn:aws:lambda:us-east-1:254067382080:layer:splunk-apm:97
    arn:aws:lambda:eu-central-1:254067382080:layer:splunk-apm:97
    arn:aws:lambda:us-west-1:254067382080:layer:splunk-apm:97
    arn:aws:lambda:us-west-2:254067382080:layer:splunk-apm:97
    arn:aws:lambda:ap-south-1:254067382080:layer:splunk-apm:96
    arn:aws:lambda:ap-northeast-1:254067382080:layer:splunk-apm:96
    arn:aws:lambda:ap-northeast-2:254067382080:layer:splunk-apm:96
    arn:aws:lambda:ap-southeast-1:254067382080:layer:splunk-apm:96
    arn:aws:lambda:ap-southeast-2:254067382080:layer:splunk-apm:95
    arn:aws:lambda:ca-central-1:254067382080:layer:splunk-apm:95
    arn:aws:lambda:eu-west-1:254067382080:layer:splunk-apm:95
    arn:aws:lambda:eu-west-2:254067382080:layer:splunk-apm:95
    arn:aws:lambda:eu-west-3:254067382080:layer:splunk-apm:95
    arn:aws:lambda:eu-north-1:254067382080:layer:splunk-apm:95
    arn:aws:lambda:sa-east-1:254067382080:layer:splunk-apm:95
    arn:aws:lambda:eu-south-1:254067382080:layer:splunk-apm:42
    arn:aws:lambda:ap-northeast-3:254067382080:layer:splunk-apm:42
    arn:aws:lambda:ap-east-1:254067382080:layer:splunk-apm:42
    arn:aws:lambda:af-south-1:254067382080:layer:splunk-apm:42
    arn:aws:lambda:me-south-1:254067382080:layer:splunk-apm:42
    </pre>
    
  4. Paste the selected ARN in the Specify an ARN field and select Add.

  5. Check that the Splunk layer appears in the Layers table.

Note

You can automate the update of the Lambda layer using the AWS CLI or other automation tools.

Configure the Splunk OpenTelemetry Lambda layer ๐Ÿ”—

Follow these steps to add the required configuration for the Splunk OpenTelemetry Lambda layer:

  1. In the AWS Lambda console, open the function that you are instrumenting.

  2. Navigate to Configuration, then Environment variables.

  3. Select Edit.

  4. Add each of the following environment variables by selecting Add environment variable:

    Environment variable

    Description

    SPLUNK_REALM

    To find the realm of your Splunk Observability Cloud account, open the navigation menu in Splunk Observability Cloud, select Settings, and select your username. The realm name appears in the Organizations section.

    SPLUNK_ACCESS_TOKEN

    A Splunk authentication token that lets exporters send data directly to Splunk Observability Cloud. See Authentication token. To send data to a Splunk OTel Collector in EC2, see Send spans to a Collector in data forwarding mode.

    AWS_LAMBDA_EXEC_WRAPPER

    Set the value for the AWS_LAMBDA_EXEC_WRAPPER environment variable:

    # Select the most appropriate value
    
    # Wraps regular handlers that implement RequestHandler
    /opt/otel-handler
    
    # Same as otel-handler, but proxied through API Gateway,
    # with HTTP context propagation activated
    /opt/otel-proxy-handler
    
    # Wraps streaming handlers that implement RequestStreamHandler
    /opt/otel-stream-handler
    

    Note

    Only AWS SDK v2 instrumentation is activated by default. To instrument other libraries, modify your code to include the corresponding library instrumentation from the OpenTelemetry Java SDK.

    OTEL_SERVICE_NAME

    The name of your service.

    (Optional) OTEL_RESOURCE_ATTRIBUTES

    Define the name of the deployment environment of your function by setting this environment variable to deployment.environment=<name-of-your-environment>.

  5. Select Save and check that the environment variables appear in the table.

To configure the mode of metric ingest, see Metrics configuration.

Note

By default, the layer sends telemetry to a Collector instance on localhost.

Instrument Go functions in AWS Lambda ๐Ÿ”—

To instrument a Go function in AWS Lambda for Splunk APM, follow these steps:

  1. Run the following commands to install the otellambda and the Splunk OTel Go distribution:

    go get -u go.opentelemetry.io/contrib/instrumentation/github.com/aws/aws-lambda-go/otellambda
    go get -u github.com/signalfx/splunk-otel-go/distro
    
  2. Create a wrapper for the OpenTelemetry instrumentation in your functionโ€™s code. For example:

    package main
    
    import (
       "context"
       "fmt"
    
       "github.com/aws/aws-lambda-go/lambda"
       "github.com/signalfx/splunk-otel-go/distro"
       "go.opentelemetry.io/contrib/instrumentation/github.com/aws/aws-lambda-go/otellambda"
       "go.opentelemetry.io/otel"
    )
    
    func main() {
       distro.Run()
       flusher := otel.GetTracerProvider().(otellambda.Flusher)
       lambda.Start(otellambda.InstrumentHandler(HandleRequest, otellambda.WithFlusher(flusher)))
    }
    
    type MyEvent struct {
       Name string `json:"name"`
    }
    
    func HandleRequest(ctx context.Context, name MyEvent) (string, error) {
       return fmt.Sprintf("Hello %s!", name.Name), nil
    }
    

Serverless Framework support ๐Ÿ”—

Some features of the Serverless Framework might impact OpenTelemetry tracing of Python Lambda functions.

Python libraries compression ๐Ÿ”—

The zip feature of pythonRequirements allows packing and deploying Lambda dependencies as compressed files. To instrument packages compressed using the Serverless Framework, set the SPLUNK_LAMBDA_SLS_ZIP environment variable to true. For more information, see https://github.com/serverless/serverless-python-requirements#dealing-with-lambdas-size-limitations on GitHub.

Slim feature ๐Ÿ”—

The Slim feature reduces the size of Lambda packages by removing some files, including dist-info folders. Some of the files removed by the Slim feature are required by the OpenTelemetry Python autoinstrumentation. Deactivate the slim option in your serverless.yml file or define custom slimPatterns. For more information, see https://github.com/serverless/serverless-python-requirements#slim-package on GitHub.

Send spans to a Collector in data forwarding mode ๐Ÿ”—

You can send data from multiple instrumented functions to a Collector running in data forwarding (gateway) mode in EC2. This helps aggregate data and reduce load.

To send spans from the instrumented Lambda function to the Collector gateway running in EC2, do the following:

  1. Deploy the Collector in Gateway mode in a service your Lambda can reach, for example EC2. See Data forwarding (gateway) mode.

  2. Make sure that the Lambda functions you want to instrument can reach the Collector gateway in EC2 and are in the same VPC.

  3. Instrument the functions. See Install the AWS Lambda layer.

  4. Navigate to Configuration > Environment variables, then select Edit.

  5. Add the following environment variables to your instrumented functions:

    • OTEL_EXPORTER_OTLP_TRACES_ENDPOINT with the address of the EC2 instance that runs the gateway, for example `` 10.0.0.123:4317``

    • OTEL_TRACES_EXPORTER with the value otlp

    • OTLP_EXPORTER_OTLP_TRACES_PROTOCOL with the value http/protobuf

    • SPLUNK_LAMBDA_LOCAL_COLLECTOR_ENABLED with the value false

  6. If youโ€™ve already set the access token and realm in the Collector configuration, delete the SPLUNK_ACCESS_TOKEN and SPLUNK_REALM environment variables.

Send spans directly to Splunk Observability Cloud ๐Ÿ”—

By default, the Splunk OpenTelemetry Lambda layer sends telemetry to a Collector running alongside the Lambda.

To send spans directly to Splunk Observability Cloud from an AWS Lambda function instrumented using the Splunk Lambda layer add the following environment variables:

  • OTEL_EXPORTER_OTLP_TRACES_PROTOCOL with the value http/protobuf

  • OTEL_EXPORTER_OTLP_TRACES_ENDPOINT with the value https://ingest.<realm>.signalfx.com/v2/trace/otlp, substituting <realm> with the name of your organizationโ€™s realm.

Check that data appears in Splunk Observability Cloud ๐Ÿ”—

Each time the AWS Lambda function runs, trace and metric data appears in Splunk Observability Cloud. If no data appears, see Troubleshoot the Splunk OpenTelemetry Lambda Layer.