Instrument your AWS Lambda function using a Collector gateway in EC2 π
You can automatically instrument your AWS Lambda function for Splunk Observability Cloud using a layer for your runtime and a Collector gateway in EC2. Using a Collector gateway helps aggregate data and reduce load.
Follow these instructions to install the Splunk OpenTelemetry Lambda layer for Java, Python, or Node.js functions. See Instrument your AWS Lambda function for Splunk Observability Cloud for the all-in-one default deployment instructions.
Check compatibility and requirements π
The Splunk OpenTelemetry Lambda layer supports the following runtimes in AWS Lambda:
Java 8.al2, 11, 17, and 21
Python 3.8 and higher
Node.js 14 and higher
Go 1.21
The AWS Lambda layer requires 90 MB on-disk in standard x86_64 systems.
For more information, search for Lambda runtimes and other system requirements on the AWS documentation website.
Install the AWS Lambda layer for your language π
Follow these steps to instrument your function using the Splunk OpenTelemetry Lambda layer:
In the AWS Lambda console, select the function that you want to instrument.
In the Layers section, select Add a layer, then select Specify an ARN.
Copy the Amazon Resource Name (ARN) for the region of your AWS Lambda function from the list that matches your architecture:
<h3>Splunk OpenTelemetry Lambda Java Layer</h3> <pre> arn:aws:lambda:us-east-2:254067382080:layer:splunk-apm-java:137 arn:aws:lambda:us-east-1:254067382080:layer:splunk-apm-java:11 arn:aws:lambda:eu-central-1:254067382080:layer:splunk-apm-java:11 arn:aws:lambda:us-west-1:254067382080:layer:splunk-apm-java:11 arn:aws:lambda:us-west-2:254067382080:layer:splunk-apm-java:11 arn:aws:lambda:ap-south-1:254067382080:layer:splunk-apm-java:11 arn:aws:lambda:ap-northeast-1:254067382080:layer:splunk-apm-java:11 arn:aws:lambda:ap-northeast-2:254067382080:layer:splunk-apm-java:11 arn:aws:lambda:ap-southeast-1:254067382080:layer:splunk-apm-java:11 arn:aws:lambda:ap-southeast-2:254067382080:layer:splunk-apm-java:11 arn:aws:lambda:ca-central-1:254067382080:layer:splunk-apm-java:11 arn:aws:lambda:eu-west-1:254067382080:layer:splunk-apm-java:11 arn:aws:lambda:eu-west-2:254067382080:layer:splunk-apm-java:11 arn:aws:lambda:eu-west-3:254067382080:layer:splunk-apm-java:11 arn:aws:lambda:eu-north-1:254067382080:layer:splunk-apm-java:11 arn:aws:lambda:sa-east-1:254067382080:layer:splunk-apm-java:11 arn:aws:lambda:eu-south-1:254067382080:layer:splunk-apm-java:11 arn:aws:lambda:ap-northeast-3:254067382080:layer:splunk-apm-java:11 arn:aws:lambda:ap-east-1:254067382080:layer:splunk-apm-java:11 arn:aws:lambda:af-south-1:254067382080:layer:splunk-apm-java:11 arn:aws:lambda:me-south-1:254067382080:layer:splunk-apm-java:11 </pre>
<h3>Splunk OpenTelemetry Lambda Java Layer (arm64)</h3> <pre> arn:aws:lambda:us-east-2:254067382080:layer:splunk-apm-java-arm:11 arn:aws:lambda:us-east-1:254067382080:layer:splunk-apm-java-arm:11 arn:aws:lambda:eu-central-1:254067382080:layer:splunk-apm-java-arm:11 arn:aws:lambda:us-west-1:254067382080:layer:splunk-apm-java-arm:11 arn:aws:lambda:us-west-2:254067382080:layer:splunk-apm-java-arm:11 arn:aws:lambda:ap-south-1:254067382080:layer:splunk-apm-java-arm:11 arn:aws:lambda:ap-northeast-1:254067382080:layer:splunk-apm-java-arm:11 arn:aws:lambda:ap-northeast-2:254067382080:layer:splunk-apm-java-arm:11 arn:aws:lambda:ap-southeast-1:254067382080:layer:splunk-apm-java-arm:11 arn:aws:lambda:ap-southeast-2:254067382080:layer:splunk-apm-java-arm:11 arn:aws:lambda:ca-central-1:254067382080:layer:splunk-apm-java-arm:11 arn:aws:lambda:eu-west-1:254067382080:layer:splunk-apm-java-arm:11 arn:aws:lambda:eu-west-2:254067382080:layer:splunk-apm-java-arm:11 arn:aws:lambda:eu-west-3:254067382080:layer:splunk-apm-java-arm:11 arn:aws:lambda:eu-north-1:254067382080:layer:splunk-apm-java-arm:11 arn:aws:lambda:sa-east-1:254067382080:layer:splunk-apm-java-arm:11 arn:aws:lambda:eu-south-1:254067382080:layer:splunk-apm-java-arm:11 arn:aws:lambda:ap-northeast-3:254067382080:layer:splunk-apm-java-arm:11 arn:aws:lambda:ap-east-1:254067382080:layer:splunk-apm-java-arm:11 arn:aws:lambda:af-south-1:254067382080:layer:splunk-apm-java-arm:11 arn:aws:lambda:me-south-1:254067382080:layer:splunk-apm-java-arm:11 </pre>
<h3>Splunk OpenTelemetry Lambda Node.js Layer</h3> <pre> arn:aws:lambda:us-east-2:254067382080:layer:splunk-apm-js:137 arn:aws:lambda:us-east-1:254067382080:layer:splunk-apm-js:11 arn:aws:lambda:eu-central-1:254067382080:layer:splunk-apm-js:11 arn:aws:lambda:us-west-1:254067382080:layer:splunk-apm-js:11 arn:aws:lambda:us-west-2:254067382080:layer:splunk-apm-js:11 arn:aws:lambda:ap-south-1:254067382080:layer:splunk-apm-js:11 arn:aws:lambda:ap-northeast-1:254067382080:layer:splunk-apm-js:11 arn:aws:lambda:ap-northeast-2:254067382080:layer:splunk-apm-js:11 arn:aws:lambda:ap-southeast-1:254067382080:layer:splunk-apm-js:11 arn:aws:lambda:ap-southeast-2:254067382080:layer:splunk-apm-js:11 arn:aws:lambda:ca-central-1:254067382080:layer:splunk-apm-js:11 arn:aws:lambda:eu-west-1:254067382080:layer:splunk-apm-js:11 arn:aws:lambda:eu-west-2:254067382080:layer:splunk-apm-js:11 arn:aws:lambda:eu-west-3:254067382080:layer:splunk-apm-js:11 arn:aws:lambda:eu-north-1:254067382080:layer:splunk-apm-js:11 arn:aws:lambda:sa-east-1:254067382080:layer:splunk-apm-js:11 arn:aws:lambda:eu-south-1:254067382080:layer:splunk-apm-js:11 arn:aws:lambda:ap-northeast-3:254067382080:layer:splunk-apm-js:11 arn:aws:lambda:ap-east-1:254067382080:layer:splunk-apm-js:11 arn:aws:lambda:af-south-1:254067382080:layer:splunk-apm-js:11 arn:aws:lambda:me-south-1:254067382080:layer:splunk-apm-js:11 </pre>
<h3>Splunk OpenTelemetry Lambda Node.js Layer (arm64)</h3> <pre> arn:aws:lambda:us-east-2:254067382080:layer:splunk-apm-js-arm:11 arn:aws:lambda:us-east-1:254067382080:layer:splunk-apm-js-arm:11 arn:aws:lambda:eu-central-1:254067382080:layer:splunk-apm-js-arm:11 arn:aws:lambda:us-west-1:254067382080:layer:splunk-apm-js-arm:11 arn:aws:lambda:us-west-2:254067382080:layer:splunk-apm-js-arm:11 arn:aws:lambda:ap-south-1:254067382080:layer:splunk-apm-js-arm:11 arn:aws:lambda:ap-northeast-1:254067382080:layer:splunk-apm-js-arm:11 arn:aws:lambda:ap-northeast-2:254067382080:layer:splunk-apm-js-arm:11 arn:aws:lambda:ap-southeast-1:254067382080:layer:splunk-apm-js-arm:11 arn:aws:lambda:ap-southeast-2:254067382080:layer:splunk-apm-js-arm:11 arn:aws:lambda:ca-central-1:254067382080:layer:splunk-apm-js-arm:11 arn:aws:lambda:eu-west-1:254067382080:layer:splunk-apm-js-arm:11 arn:aws:lambda:eu-west-2:254067382080:layer:splunk-apm-js-arm:11 arn:aws:lambda:eu-west-3:254067382080:layer:splunk-apm-js-arm:11 arn:aws:lambda:eu-north-1:254067382080:layer:splunk-apm-js-arm:11 arn:aws:lambda:sa-east-1:254067382080:layer:splunk-apm-js-arm:11 arn:aws:lambda:eu-south-1:254067382080:layer:splunk-apm-js-arm:11 arn:aws:lambda:ap-northeast-3:254067382080:layer:splunk-apm-js-arm:11 arn:aws:lambda:ap-east-1:254067382080:layer:splunk-apm-js-arm:11 arn:aws:lambda:af-south-1:254067382080:layer:splunk-apm-js-arm:11 arn:aws:lambda:me-south-1:254067382080:layer:splunk-apm-js-arm:11 </pre>
<h3>Splunk OpenTelemetry Lambda Python Layer</h3> <pre> arn:aws:lambda:us-east-2:254067382080:layer:splunk-apm-python:137 arn:aws:lambda:us-east-1:254067382080:layer:splunk-apm-python:11 arn:aws:lambda:eu-central-1:254067382080:layer:splunk-apm-python:11 arn:aws:lambda:us-west-1:254067382080:layer:splunk-apm-python:11 arn:aws:lambda:us-west-2:254067382080:layer:splunk-apm-python:11 arn:aws:lambda:ap-south-1:254067382080:layer:splunk-apm-python:11 arn:aws:lambda:ap-northeast-1:254067382080:layer:splunk-apm-python:11 arn:aws:lambda:ap-northeast-2:254067382080:layer:splunk-apm-python:11 arn:aws:lambda:ap-southeast-1:254067382080:layer:splunk-apm-python:11 arn:aws:lambda:ap-southeast-2:254067382080:layer:splunk-apm-python:11 arn:aws:lambda:ca-central-1:254067382080:layer:splunk-apm-python:11 arn:aws:lambda:eu-west-1:254067382080:layer:splunk-apm-python:11 arn:aws:lambda:eu-west-2:254067382080:layer:splunk-apm-python:11 arn:aws:lambda:eu-west-3:254067382080:layer:splunk-apm-python:11 arn:aws:lambda:eu-north-1:254067382080:layer:splunk-apm-python:11 arn:aws:lambda:sa-east-1:254067382080:layer:splunk-apm-python:11 arn:aws:lambda:eu-south-1:254067382080:layer:splunk-apm-python:11 arn:aws:lambda:ap-northeast-3:254067382080:layer:splunk-apm-python:11 arn:aws:lambda:ap-east-1:254067382080:layer:splunk-apm-python:11 arn:aws:lambda:af-south-1:254067382080:layer:splunk-apm-python:11 arn:aws:lambda:me-south-1:254067382080:layer:splunk-apm-python:11 </pre>
Note
For Python functions using the Serverless Framework, see Serverless Framework support.
<h3>Splunk OpenTelemetry Lambda Python Layer (arm64)</h3> <pre> arn:aws:lambda:us-east-2:254067382080:layer:splunk-apm-python-arm:11 arn:aws:lambda:us-east-1:254067382080:layer:splunk-apm-python-arm:11 arn:aws:lambda:eu-central-1:254067382080:layer:splunk-apm-python-arm:11 arn:aws:lambda:us-west-1:254067382080:layer:splunk-apm-python-arm:11 arn:aws:lambda:us-west-2:254067382080:layer:splunk-apm-python-arm:11 arn:aws:lambda:ap-south-1:254067382080:layer:splunk-apm-python-arm:11 arn:aws:lambda:ap-northeast-1:254067382080:layer:splunk-apm-python-arm:11 arn:aws:lambda:ap-northeast-2:254067382080:layer:splunk-apm-python-arm:11 arn:aws:lambda:ap-southeast-1:254067382080:layer:splunk-apm-python-arm:11 arn:aws:lambda:ap-southeast-2:254067382080:layer:splunk-apm-python-arm:11 arn:aws:lambda:ca-central-1:254067382080:layer:splunk-apm-python-arm:11 arn:aws:lambda:eu-west-1:254067382080:layer:splunk-apm-python-arm:11 arn:aws:lambda:eu-west-2:254067382080:layer:splunk-apm-python-arm:11 arn:aws:lambda:eu-west-3:254067382080:layer:splunk-apm-python-arm:11 arn:aws:lambda:eu-north-1:254067382080:layer:splunk-apm-python-arm:11 arn:aws:lambda:sa-east-1:254067382080:layer:splunk-apm-python-arm:11 arn:aws:lambda:eu-south-1:254067382080:layer:splunk-apm-python-arm:11 arn:aws:lambda:ap-northeast-3:254067382080:layer:splunk-apm-python-arm:11 arn:aws:lambda:ap-east-1:254067382080:layer:splunk-apm-python-arm:11 arn:aws:lambda:af-south-1:254067382080:layer:splunk-apm-python-arm:11 arn:aws:lambda:me-south-1:254067382080:layer:splunk-apm-python-arm:11 </pre>
Note
For Python functions using the Serverless Framework, see Serverless Framework support.
Paste the selected ARN in the Specify an ARN field and select Add.
Check that the Splunk layer appears in the Layers table.
(Optional) Repeat the previous steps to install AWS Lambda metrics extension layer for Splunk Infrastructure Monitoring.
<h3>Splunk Metrics Extension Layer</h3> <pre> arn:aws:lambda:us-east-2:254067382080:layer:splunk-lambda-metrics:137 arn:aws:lambda:us-east-1:254067382080:layer:splunk-lambda-metrics:11 arn:aws:lambda:eu-central-1:254067382080:layer:splunk-lambda-metrics:11 arn:aws:lambda:us-west-1:254067382080:layer:splunk-lambda-metrics:11 arn:aws:lambda:us-west-2:254067382080:layer:splunk-lambda-metrics:11 arn:aws:lambda:ap-south-1:254067382080:layer:splunk-lambda-metrics:11 arn:aws:lambda:ap-northeast-1:254067382080:layer:splunk-lambda-metrics:11 arn:aws:lambda:ap-northeast-2:254067382080:layer:splunk-lambda-metrics:11 arn:aws:lambda:ap-southeast-1:254067382080:layer:splunk-lambda-metrics:11 arn:aws:lambda:ap-southeast-2:254067382080:layer:splunk-lambda-metrics:11 arn:aws:lambda:ca-central-1:254067382080:layer:splunk-lambda-metrics:11 arn:aws:lambda:eu-west-1:254067382080:layer:splunk-lambda-metrics:11 arn:aws:lambda:eu-west-2:254067382080:layer:splunk-lambda-metrics:11 arn:aws:lambda:eu-west-3:254067382080:layer:splunk-lambda-metrics:11 arn:aws:lambda:eu-north-1:254067382080:layer:splunk-lambda-metrics:11 arn:aws:lambda:sa-east-1:254067382080:layer:splunk-lambda-metrics:11 arn:aws:lambda:eu-south-1:254067382080:layer:splunk-lambda-metrics:11 arn:aws:lambda:ap-northeast-3:254067382080:layer:splunk-lambda-metrics:11 arn:aws:lambda:ap-east-1:254067382080:layer:splunk-lambda-metrics:11 arn:aws:lambda:af-south-1:254067382080:layer:splunk-lambda-metrics:11 arn:aws:lambda:me-south-1:254067382080:layer:splunk-lambda-metrics:11 </pre>
<h3>Splunk Metrics Extension Layer (arm64)</h3> <pre> arn:aws:lambda:us-east-2:254067382080:layer:splunk-lambda-metrics-arm:11 arn:aws:lambda:us-east-1:254067382080:layer:splunk-lambda-metrics-arm:11 arn:aws:lambda:eu-central-1:254067382080:layer:splunk-lambda-metrics-arm:11 arn:aws:lambda:us-west-1:254067382080:layer:splunk-lambda-metrics-arm:11 arn:aws:lambda:us-west-2:254067382080:layer:splunk-lambda-metrics-arm:11 arn:aws:lambda:ap-south-1:254067382080:layer:splunk-lambda-metrics-arm:11 arn:aws:lambda:ap-northeast-1:254067382080:layer:splunk-lambda-metrics-arm:11 arn:aws:lambda:ap-northeast-2:254067382080:layer:splunk-lambda-metrics-arm:11 arn:aws:lambda:ap-southeast-1:254067382080:layer:splunk-lambda-metrics-arm:11 arn:aws:lambda:ap-southeast-2:254067382080:layer:splunk-lambda-metrics-arm:11 arn:aws:lambda:ca-central-1:254067382080:layer:splunk-lambda-metrics-arm:11 arn:aws:lambda:eu-west-1:254067382080:layer:splunk-lambda-metrics-arm:11 arn:aws:lambda:eu-west-2:254067382080:layer:splunk-lambda-metrics-arm:11 arn:aws:lambda:eu-west-3:254067382080:layer:splunk-lambda-metrics-arm:11 arn:aws:lambda:eu-north-1:254067382080:layer:splunk-lambda-metrics-arm:11 arn:aws:lambda:sa-east-1:254067382080:layer:splunk-lambda-metrics-arm:11 arn:aws:lambda:eu-south-1:254067382080:layer:splunk-lambda-metrics-arm:11 arn:aws:lambda:ap-northeast-3:254067382080:layer:splunk-lambda-metrics-arm:11 arn:aws:lambda:ap-east-1:254067382080:layer:splunk-lambda-metrics-arm:11 arn:aws:lambda:af-south-1:254067382080:layer:splunk-lambda-metrics-arm:11 arn:aws:lambda:me-south-1:254067382080:layer:splunk-lambda-metrics-arm:11 </pre>
Note
You can automate the update of the Lambda layers using the AWS CLI or other automation tools.
Configure the Splunk OpenTelemetry Lambda layer π
Follow these steps to add the required configuration for the Splunk OpenTelemetry Lambda layer:
In the AWS Lambda console, open the function that you are instrumenting.
Navigate to Configuration, then Environment variables.
Select Edit.
Add each of the following environment variables by selecting Add environment variable:
Environment variable
Description
SPLUNK_REALM
To find your Splunk realm, see Note about realms.
SPLUNK_ACCESS_TOKEN
A Splunk authentication token that lets exporters send data directly to Splunk Observability Cloud. See Authentication token. To send data to a Splunk OTel Collector in EC2, see Set up a Collector gateway in EC2.
AWS_LAMBDA_EXEC_WRAPPER
Set the value for the environment variable given your programming language:
# Select the most appropriate value # Wraps regular handlers that implement RequestHandler /opt/otel-handler # Same as otel-handler, but proxied through API Gateway, # with HTTP context propagation activated /opt/otel-proxy-handler # Wraps streaming handlers that implement RequestStreamHandler /opt/otel-stream-handler
Note
Only AWS SDK v2 instrumentation is activated by default. To instrument other libraries, modify your code to include the corresponding library instrumentation from the OpenTelemetry Java SDK.
/opt/otel-instrument
/opt/nodejs-otel-handler
OTEL_SERVICE_NAME
The name of your service.
(Optional)
OTEL_RESOURCE_ATTRIBUTES
Define the name of the deployment environment of your function by setting this environment variable to
deployment.environment=<name-of-your-environment>
.Select Save and check that the environment variables appear in the table.
To configure the mode of metric ingest, see Metrics configuration.
Note
By default, the layer sends telemetry to a Collector instance on localhost using the Collector layer. If you donβt want to use a local Collector, you must specify the address of a Collector in data forwarding mode. See Set up a Collector gateway in EC2.
Set up a Collector gateway in EC2 π
To send spans and metrics from the instrumented AWS Lambda function to the Collector gateway running in EC2, do the following:
Deploy the Collector in gateway mode in a service your function can reach, for example EC2. See Data forwarding (gateway) mode.
Make sure that the AWS Lambda functions you want to instrument can reach the Collector gateway in EC2 and are in the same VPC.
Navigate to Configuration, then Environment variables, then select Edit.
Add the following environment variables to your instrumented functions:
OTEL_EXPORTER_OTLP_TRACES_ENDPOINT
with the address of the EC2 instance that runs the gateway, for examplehttp://10.0.0.123:4318/v1/traces
OTEL_TRACES_EXPORTER
with the valueotlp
OTLP_EXPORTER_OTLP_TRACES_PROTOCOL
with the valuehttp/protobuf
SPLUNK_LAMBDA_LOCAL_COLLECTOR_ENABLED
with the valuefalse
SPLUNK_METRICS_ENDPOINT
with the address of the EC2 instance that runs the gateway, for examplehttp://10.0.0.123:9943
If youβve already set the access token and realm in the Collector configuration, delete the
SPLUNK_ACCESS_TOKEN
andSPLUNK_REALM
environment variables.
Check that data appears in Splunk Observability Cloud π
Each time the AWS Lambda function runs, trace and metric data appears in Splunk Observability Cloud. If no data appears, see Troubleshoot the Splunk OpenTelemetry Lambda layer.