LWA + OpenTelemetry collector + zip Lambda deployment - is it possible?

[tmokmss]

Hi, can we use LWA and OpenTelemetry collector together with zip Lambda deployment? I am afraid that we cannot because LWA and Otel collector both requires AWS_LAMBDA_EXEC_WRAPPER environment variable set to their binary path, but we can only set one single path for the variable.

Do you know if there is any workaround for this? Thanks.

P.S.) I confirmed that with Container Lambda deployment, we can use LWA + Otel without a problem. We can just copy multiple Lambda extension binaries to the /opt/extensions directory.

[tmokmss]

Nevermind. I misunderstood that AWS_LAMBDA_EXEC_WRAPPER is a path to binary but it turns out that it is just a shell script (lwa, otel, nodejs). I merged the two scripts and it is now apparently working.

[tmokmss]

Hi @bnusunny I'm currently trying this configuration on Lambda Node.js runtime, but now realized that things are not such simple.

So Lambda + otel seems to rely on AwsLambdaInstrumentation module to properly instrument a Lambda function, e.g. creating a span, by "patching" the Lambda handler function. But LWA apps do not have Lambda handler, (we have run.sh instead), the module fails to apply the patch, resulting in failed instrumentation (in X-Ray I can only see Lambda lifecycle events such as Init, Overhead, etc.)

I'm still trying to find a way to make it work properly, but do you have any information on this? e.g. a successful example of LWA + Otel configuration. Thanks.

[bnusunny]

Since you are running a actual web app inside Lambda using LWA, you should instrument the web app in the normal way (such as this one), not following the Lambda function approach. I can create an example this weekend.

[tmokmss]

@bnusunny That's awesome, thanks! I think I tried the approach but it somehow didn't work; it appeared the span context was not propagated? still investigating though. I might have mesed up something so I'll wait for your test result 🙏

[bnusunny]

I got it working. Here are the configuration steps:

1. install ADOT auto instrumentation lib:

npm install @aws/aws-distro-opentelemetry-node-autoinstrumentation

2. attach ADOT layer to the function:

      Layers:
        - !Sub arn:aws:lambda:${AWS::Region}:753240598075:layer:LambdaAdapterLayerX86:24
        - !Sub arn:aws:lambda:${AWS::Region}:901920570463:layer:aws-otel-nodejs-amd64-ver-1-30-1:1

3. add configuration environment variables to the run.sh script

#!/bin/bash
export NODE_OPTIONS="--require @aws/aws-distro-opentelemetry-node-autoinstrumentation/register"
export OTEL_SERVICE_NAME=expressjs
exec node app.js

The ADOT collector batches the trace data by default. You need to invoke the functions multple times before the otel trace data show up in the x-ray.

Here is a sample trace.

[tmokmss]

Thanks! I'll give it a try 👍

[tmokmss]

Hi @bnusunny could you share the full example code when you get a chance? Currently I cannot somehow get it work but not sure about the detailed difference with your configuration e.g. bundler option etc.