AWS Lambda Problem Node.js & Latest

[SouthBelair]

Environment

SaaS (https://sentry.io/)

Steps to Reproduce

We had an existing aws lambda integration in place and noticed that some of our lambdas were missing from the configure list. In trying to correct this we created a new installation and selected an existing project and aws stack, this updated our lambdas to the "latest". The lambdas immediately began producing errors (see below).

Here's what changed:

Lambda ARN changed from arn:aws:lambda:us-east-1:943013980633:layer:SentryNodeServerlessSDK:184 to arn:aws:lambda:us-east-1:943013980633:layer:SentryNodeServerlessSDK:245

NODE_OPTIONS env value changed from -r @sentry/serverless/dist/awslambda-auto to -r @sentry/aws-serverless/cjs/awslambda-auto

We had to disable sentry on all of our node.js lambdas so that they would continue to operate, in order for us to restore sentry reporting we'll have to manually update every lambda with the old layer and NODE_options values, we have hundreds of lambdas.

node:internal/modules/cjs/loader:597 throw e; ^ Error [ERR_PACKAGE_PATH_NOT_EXPORTED]: Package subpath './cjs/awslambda-auto' is not defined by "exports" in /opt/nodejs/node_modules/@sentry/aws-serverless/package.json at new NodeError (node:internal/errors:405:5) at exportsNotFound (node:internal/modules/esm/resolve:371:10) at packageExportsResolve (node:internal/modules/esm/resolve:718:9) at resolveExports (node:internal/modules/cjs/loader:590:36) at Module._findPath (node:internal/modules/cjs/loader:664:31) at Module._resolveFilename (node:internal/modules/cjs/loader:1126:27) at Module._load (node:internal/modules/cjs/loader:981:27) at internalRequire (node:internal/modules/cjs/loader:174:19) at Module._preloadModules (node:internal/modules/cjs/loader:1569:5) at loadPreloadModules (node:internal/process/pre_execution:628:5) { code: 'ERR_PACKAGE_PATH_NOT_EXPORTED' } Node.js v18.20.2 INIT_REPORT Init Duration: 156.91 ms Phase: init Status: error Error Type: Runtime.ExitError node:internal/modules/cjs/loader:597 throw e; ^ Error [ERR_PACKAGE_PATH_NOT_EXPORTED]: Package subpath './cjs/awslambda-auto' is not defined by "exports" in /opt/nodejs/node_modules/@sentry/aws-serverless/package.json at new NodeError (node:internal/errors:405:5) at exportsNotFound (node:internal/modules/esm/resolve:371:10) at packageExportsResolve (node:internal/modules/esm/resolve:718:9) at resolveExports (node:internal/modules/cjs/loader:590:36) at Module._findPath (node:internal/modules/cjs/loader:664:31) at Module._resolveFilename (node:internal/modules/cjs/loader:1126:27) at Module._load (node:internal/modules/cjs/loader:981:27) at internalRequire (node:internal/modules/cjs/loader:174:19) at Module._preloadModules (node:internal/modules/cjs/loader:1569:5) at loadPreloadModules (node:internal/process/pre_execution:628:5) { code: 'ERR_PACKAGE_PATH_NOT_EXPORTED' } Node.js v18.20.2 INIT_REPORT Init Duration: 634.31 ms Phase: invoke Status: error Error Type: Runtime.ExitError START RequestId: e2961f2a-10cc-4980-affa-a616efaaa7b8 Version: $LATEST RequestId: e2961f2a-10cc-4980-affa-a616efaaa7b8 Error: Runtime exited with error: exit status 1 Runtime.ExitError END RequestId: e2961f2a-10cc-4980-affa-a616efaaa7b8 REPORT RequestId: e2961f2a-10cc-4980-affa-a616efaaa7b8 Duration: 655.34 ms Billed Duration: 656 ms Memory Size: 256 MB Max Memory Used: 15 MB

Expected Result

N/A

Actual Result

N/A

Product Area

Settings - Integrations

Link

No response

DSN

No response

Version

No response

[getsantry[bot]]

Assigning to @getsentry/support for routing ⏲️

[getsantry[bot]]

Routing to @getsentry/product-owners-settings-integrations for triage ⏲️

[fnfilho]

Having the same problem in production today. Had to rollback and remove our Sentry setup.

[andreiborza]

hello, sorry that you've experienced that.

Version 184 of the Lambda layer was still running the v7 version of our SDK.

While it's not a true fix for your problem, you can continue to use a v7 version of the layer. We specifically published one for 7.116.0, please see https://docs.sentry.io/platforms/javascript/guides/aws-lambda/layer/#lambda-layer-for-v7 for the ARN (unfortunately, you need to modify it yourself for your specific region).

Let me know if that helps at all.

[fnfilho]

Hey @andreiborza,

Thank you for following up. I am still not able to use AWS Lambda Layer. Ticket #12012 got me confused as well to what is the right configs to set and I can't seem to find a definitive guide and docs of which Lambda Layer Version refers to what SDK Version.

So this is what I've tried so far, all combinations of the following values. Nothing seems to work.

ARNS: arn:aws:lambda:sa-east-1:943013980633:layer:SentryNodeServerlessSDK:184 arn:aws:lambda:sa-east-1:943013980633:layer:SentryNodeServerlessSDK:240 arn:aws:lambda:sa-east-1:943013980633:layer:SentryNodeServerlessSDK:250 arn:aws:lambda:us-west-1:943013980633:layer:SentryNodeServerlessSDKv7:1

NODE_OPTIONS --require @sentry/aws-serverless/cjs/awslambda-auto --require @sentry/aws-serverless/dist/awslambda-auto --require @sentry/aws-serverless/awslambda-auto

SENTRY_DSN https://[email protected]/XXX

SENTRY_TRACES_SAMPLE_RATE 1.0

[paultheurer]

Having this issue as well. Had to turn off sentry lambda layer for all of our lambdas.

Updating to the latest version of the lambda layer via the sentry UI caused this.

[andreiborza]

For arn:...:SentryNodeServerlessSDK:235 and lower, and arn:...:SentryNodeServerlessSDKv7:1 (these use the v7 version of our SDK) the NODE_OPTIONS should include --require @sentry/serverless/dist/awslambda-auto

[fnfilho]

Thank you @andreiborza I can confirm version 235 and NODE_OPTIONS @sentry/serverless/dist/awslambda-auto builds and runs correctly.

I also have set SENTRY_DSN: https://[email protected]/xxx and SENTRY_TRACES_SAMPLE_RATE: 1.0

With that said, neither errors nor performance data is being sent to Sentry. Is there a way to debug it better?

[andreiborza]

@fnfilho try setting debug: true and checking your aws logs. If you don't spot anything, please paste it here.

[fnfilho]

Hey @andreiborza How exactly should I set debug: true? Is there an environment variable that has Sentry logs?

I did try by changing the package, but still did not work. This is what I did:

1. Downloaded the 235 version package;
2. Changed the awslambda-auto.js file, adding the debug: true parameter;
3. Uploaded a new zip file as a Custom AWS Layer (file attached);
4. Updated a function with the new custom AWS Layers that runs my own package with the parameter;
5. Ran the function

Logging than said: [Sentry] Cannot initialize SDK with debug option using a non-debug bundle.

sentry7.zip

[andreiborza]

@fnfilho you should be able to set SENTRY_DEBUG=1 in the env variables you provide to your lambda function. Let me know if you can't and I'll have another look.

[fnfilho]

Thank you for following up, @andreiborza. I rolled back to package 235 and had the following env variables set:

NODE_OPTIONS: --require @sentry/serverless/dist/awslambda-auto --enable-source-maps --stack-trace-limit=1000
SENTRY_DEBUG: 1
SENTRY_DSN: https://[email protected]/xxx
SENTRY_TRACES_SAMPLE_RATE: 1.0

Than I forced an error to happen by using throw new Error("Sentry test"). No erros captured by Sentry and no logs in AWS about Sentry neither.

[fnfilho]

I tried updating to the new version 252 (arn: arn:aws:lambda:sa-east-1:943013980633:layer:SentryNodeServerlessSDK:252). Can confirm the problem persists.

Error: Error [ERR_PACKAGE_PATH_NOT_EXPORTED]: Package subpath './cjs/awslambda-auto' is not defined by "exports" in /opt/nodejs/node_modules/@sentry/aws-serverless/package.json

Env variables: NODE_OPTIONS: -r @sentry/aws-serverless/cjs/awslambda-auto SENTRY_DSN: https://[email protected]/xxx SENTRY_TRACES_SAMPLE_RATE: 1.0

[andreiborza]

@fnfilho that's because all the versions above 235 use the new export via --require @sentry/aws-serverless/awslambda-auto.

[fnfilho]

Hey @andreiborza, I get you have too many tickets to go through and you might not seen all the comments I did and just checked the latest one. So I'll try to summarize it all here.

TRYING WITH V7 ARN: arn:aws:lambda:sa-east-1:943013980633:layer:SentryNodeServerlessSDK:235 NODE_OPTIONS: --require @sentry/serverless/dist/awslambda-auto --enable-source-maps --stack-trace-limit=1000 SENTRY_DEBUG: 1 SENTRY_DSN: https://[email protected]/xxx SENTRY_TRACES_SAMPLE_RATE: 1.0

Package Builds Ok. No erros on CloudWatch. No debug messages either. But when I throw an Error in code, this is not captured by Sentry. There are no logs from Sentry at all in CloudWatch.

TRYING WITH V8 ARN: arn:aws:lambda:sa-east-1:943013980633:layer:SentryNodeServerlessSDK:252 NODE_OPTIONS: --require @sentry/aws-serverless/awslambda-auto --enable-source-maps --stack-trace-limit=1000 SENTRY_DEBUG: 1 SENTRY_DSN: https://[email protected]/xxx SENTRY_TRACES_SAMPLE_RATE: 1.0

Package won't build. There are a few logs in CloudWatch:

/opt/nodejs/node_modules/@sentry/aws-serverless/build/npm/cjs/index-C86OzNuF.js:2
Node.js v18.20.3
INIT_REPORT Init Duration: 382.11 ms Phase: invoke Status: error Error Type: Runtime.ExitError
START RequestId: 042cf6a4-429a-467a-b727-7e84bf8a9b44 Version: $LATEST
RequestId: 042cf6a4-429a-467a-b727-7e84bf8a9b44 Error: Runtime exited with error: exit status 1 Runtime.ExitError
END RequestId: 042cf6a4-429a-467a-b727-7e84bf8a9b44

It also seems to log the Sentry code. Attached full log export. logs.txt

Side note: when setting up the layer via the Sentry Dashboard in Integrations, it is adding the wrong env variable, it sets --require @sentry/aws-serverless/cjs/awslambda-auto instead of the right one --require @sentry/aws-serverless/awslambda-auto

[andreiborza]

Thank you for the rundown. I'll try to reproduce this tomorrow when I have a bit more time.

As for the Sentry Dashboard, thanks! This should be rolling out soon: https://github.com/getsentry/sentry/pull/73267

[andreiborza]

We just shipped a new release that fixes an issue with underlying opentelemetry dependencies. The new layer is SentryNodeServerlessSDK:255 for v8 and SentryNodeServerlessSDKv7:3 for v7.

Please note, for v7 the way you import the package and wrap your lambda handler is different, I updated the docs with code snippets: https://docs.sentry.io/platforms/javascript/guides/aws-lambda/layer/#lambda-layer-for-v7

@fnfilho please try updating your layer to 255 if you are using v8 of sentry, otherwise 3 for the v7 layer.

[webjay]

I still don't see a layer added with SentryNodeServerlessSDK:255

Another thing, should I set NODE_OPTIONS to "-r @sentry/aws-serverless/awslambda-auto"?

[andreiborza]

@webjay the layer version depends on your aws region. Please use the dropdown here to select your region to get the appropriate ARN.

If you're using the lambda layer as described in the docs linked, you won't need node options. If you're using the (deprecated) container you'll need NODE_OPTIONS="-r @sentry/aws-serverless/awslambda-auto" as described on the docs page for the container.

[webjay]

@andreiborza This is my setup:

export function SentryStack({ app, stack }: StackContext) {
  const sentry = LayerVersion.fromLayerVersionArn(
    stack,
    "SentryLayer",
    `arn:aws:lambda:${app.region}:943013980633:layer:SentryNodeServerlessSDK:255`,
  );

  stack.addDefaultFunctionLayers([sentry]);
  stack.addDefaultFunctionEnv({
    SENTRY_DSN: process.env.SENTRY_DSN ?? "",
    SENTRY_TRACES_SAMPLE_RATE: "1.0",
    SENTRY_DEBUG: "1",
  });

  return { sentry };
}

However I still don't see the layer attached. Does it support Node 18 & 20?

[fnfilho]

Thank you for the update @andreiborza

Trying with V8 seems to have improved ARN: arn:aws:lambda:sa-east-1:943013980633:layer:SentryNodeServerlessSDK:255 NODE_OPTIONS: --require @sentry/aws-serverless/awslambda-auto --enable-source-maps --stack-trace-limit=1000 SENTRY_DEBUG: 1 SENTRY_DSN: https://[email protected]/xxx SENTRY_TRACES_SAMPLE_RATE: 1.0

Package now builds ok and no errors are logged. But when I force my functions to die with throw new Error('Sentry test'), it doesn't gets sent to Sentry. Setting SENTRY_DEBUG to 1 didn't add any logs that could help understanding what is going on.

[webjay]

My challenge was with SST. Changing stack.addDefaultFunctionLayers([sentryLayer]); to app.addDefaultFunctionLayers([sentryLayer]); fixed it and I can now see the layer.

[andreiborza]

@fnfilho please paste a snippet of your function.

For reference, here's the one I'm testing with based on AWS's blank-nodejs sample app that works for me with the env variables you provided and my dsn.

const Sentry = require("@sentry/aws-serverless");

const AWSXRay = require('aws-xray-sdk-core');
const { LambdaClient, GetAccountSettingsCommand } = require('@aws-sdk/client-lambda');

// Create client outside of handler to reuse
const lambda = AWSXRay.captureAWSv3Client(new LambdaClient());

// Handler
exports.handler = Sentry.wrapHandler(async function(event, context) {
    console.log('options', JSON.stringify(Sentry.getClient().getOptions()));
    
    event.Records.forEach(record => {
        console.log(record.body);
    });

    console.log('## ENVIRONMENT VARIABLES: ' + serialize(process.env));
    console.log('## CONTEXT: ' + serialize(context));
    console.log('## EVENT: ' + serialize(event));
    
    throw new Error('This should show up in sentry.')

    return getAccountSettings();
});

// Use SDK client
var getAccountSettings = function() {
    return lambda.send(new GetAccountSettingsCommand());
};

var serialize = function(object) {
    return JSON.stringify(object, null, 2);
};

[fnfilho]

Hey @andreiborza I've tried to do some extra debugging here to figure out why errors are not shown on sentry.

This is my configuration: ARN: arn:aws:lambda:sa-east-1:943013980633:layer:SentryNodeServerlessSDK:255 NODE_OPTIONS: --require @sentry/aws-serverless/awslambda-auto --enable-source-maps --stack-trace-limit=1000 SENTRY_DSN: https://[email protected]/xxx SENTRY_TRACES_SAMPLE_RATE: 1.0

This builds correctly. I than added a code to change the debug param:

const Sentry = require("@sentry/aws-serverless");
Sentry.getClient().getOptions().debug = true;
console.log("Sentry initiated.", { sentryConfig: Sentry.getClient().getOptions() });

Which prints something that looks ok:

INFO	Sentry initiated. {
  sentryConfig: {
    transport: [Function: d9],
    dsn: 'https://[email protected]/xxx',
    Ie: { sdk: [Object] },
    defaultIntegrations: [[Object]],
    integrations: [[Object]],
    autoSessionTracking: false,
    tracesSampleRate: 1,
    stackParser: [Function (anonymous)],
    platform: 'node',
    runtime: { name: 'node', version: 'v18.20.3' },
    serverName: '169.254.8.137',
    debug: true
  }
}

I than throw an error:

try {
    throw new Error("sentry testing 2");
} catch (error: any) {
    const response = Sentry.captureException(error);
    console.log("Sentry exception captured.", { response });
}

Which prints me a log with an ID that I think is the error Code from Sentry:

INFO	Sentry exception captured. { response: 'b6ca4a76e0434e048c45f357910c3b53' }

But still. No errors are tracked in Sentry at all.

[webjay]

I can add that I thought the reason why I didn't see my test errors etc was caused by us being over our limit, but I don't see any dropped errors.

The 3 errors I got in was not via the layer, but with @sentry/aws-serverless v8.11.0

So I'm in the same ⛵ as @fnfilho 👋

[lforst]

@fnfilho @webjay can you guys try calling await Sentry.flush() at the end of your handlers?

[webjay]

@lforst I tried this and still see nothing in Sentry. In the response I do see a Sentry id.

export const getAppVersionsController = async () => {
  const sentryId = Sentry.captureMessage("test message");
  await Sentry.flush();
  return createResponse({
    android: ANDROID_VERSION,
    ios: IOS_VERSION,
    sentryId,
  });
};

[fnfilho]

Hey @lforst I'm getting the same thing as @webjay

Tried that a few hours ago.

const response = Sentry.captureException(error);
await Sentry.flush();
console.log("Sentry exception captured.", { response });

I also get the "Sentry exception captured" log with a response Id. Still no bugs being capture in Sentry

[andreiborza]

captureException generates ids on the client, it's not an indicator that things went through.

My recommendation is to work backwards from the code snippet I posted here and see if it works and then gradually replace code with your own implementation.

Also note that we are still working on better support for ESM, see this issue: https://github.com/getsentry/sentry-javascript/issues/12409.

[webjay]

To make it work, I have to:

Sentry.init({
  dsn: process.env.SENTRY_DSN,
});

and Sentry.flush()

Is that how it is supposed to work @andreiborza