make AWS SDK JS v2 and v3 available with ESM in AWS Lambda

[shishkin]

Describe the bug

Nodejs lambda is not able to find AWS SDK.

Your environment

SDK version number

Tried latest v2 and latest v3 with same effect: @aws-sdk/client-s3@npm:3.48.0 aws-sdk@npm:2.1062.0

Is the issue in the browser/Node.js/ReactNative?

Node.js

Details of the browser/Node.js/ReactNative version

Node.js 14 AWS Lambda runtime

Steps to reproduce

Here is the code of the lambda:

// src/lambda.ts
import { S3 } from "aws-sdk";
import { URLSearchParams } from "node:url";
var { REGION, BUCKET_NAME } = process.env;
var s3 = new S3({ region: REGION });
var handler = async (e) => {
  console.debug("Event:", e);
  const { message } = e;
  const greeting = hello(message);
  const s3result = await s3.putObject({
    Bucket: BUCKET_NAME,
    Key: `greetings/hello.txt`,
    Tagging: new URLSearchParams({
      type: "greeting"
    }).toString(),
    ContentType: "text/plain; charset=UTF-8",
    Body: greeting
  }).promise();
  console.debug("S3 result:", s3result);
};
var hello = (message) => `Hello, ${message}!`;
export {
  handler,
  hello
};

Observed behavior

Lambda fails with error:

Error [ERR_MODULE_NOT_FOUND]: Cannot find package 'aws-sdk' imported from /var/task/index.mjs

Expected behavior

Reading the guide, I understand that aws-sdk should be already available on Nodejs runtimes.

Additional context

I'm trying to make use of newly announced ESM support in the Nodejs14 AWS Lambda runtime and make the smallest possible lambda bundle, hence the reason for not bundling AWS SDK.

[vudh1]

Hi @shishkin thanks for reaching out. Make sure that you install the package aws-sdk in the right directory. Can we know how do you install?

[shishkin]

I install SDK locally with Yarn/npm. Not sure it matters. My question is about SDK not being available on the AWS nodejs lambda runtime, which I don't install anything onto beyond copying my function code bundle. Please let me know if I'm mistaken in my assumptions.

[TimNN]

FYI: I ran into a similar problem and was able to make this work by creating and using a require function as documented here: https://nodejs.org/api/module.html#modulecreaterequirefilename

[shishkin]

@TimNN to better understand your solution, do you then const S3 = require("aws-sdk/S3") instead of importing it?

[TimNN]

Almost. Importing specific clients did not work for me, but const { EC2 } = require('aws-sdk'); does.

[shishkin]

Thanks @TimNN! I've built up on your suggestion to support TypeScript transpiling using CDK Node.js bundling:

    new NodejsFunction(this, "Lambda", {
      entry: fileURLToPath(new URL("handler.ts", import.meta.url)),
      bundling: {
        format: OutputFormat.ESM,
        target: "esnext",
        banner: `const AWS=await (async ()=>{const { createRequire }=(await import("node:module")).default;const require=createRequire(import.meta.url);return require("aws-sdk");})();`,
      },
    });

[shishkin]

@vudh1 could you please provide an update on the triage of this bug? While the workaround has been identified, lambda runtime should support ESM imports of aws-sdk natively.

[simlu]

Just ran into this as well. Weird bug and hard to understand why it's happening.

[FrancoCorleone]

Looks like importing absolute path(/var/runtime(...)) is a workaround... but an ugly one.

[dinkzilla]

Also having this issue. I think this is a bug with the lambda runtime's es module support and not a bug with the sdk itself. It's at least the fifth one I've run into at this point. I cannot recommend enough sticking with CommonJS or a transpiiler for the time being.

Is there somewhere more generic we can submit this bug for triage?

[shishkin]

@FrancoCorleone what would be the absolute path for aws-sdk package?

[FrancoCorleone]

import AWS from '/var/runtime/node_modules/aws-sdk/lib/aws.js' this works for me

[EPMatt]

Same issue here. Trying to import aws-sdk v2 within a TypeScript module compiled in ES2022, using ESM imports:

import AWS from 'aws-sdk';

The lambda fails with the error: Cannot find package 'aws-sdk' imported from /var/task/index.js\nDid you mean to import aws-sdk/lib/aws.js?

A valid workaround is to add aws-sdk as a dependency in the TypeScript module's package.json (npm i aws-sdk). This allows the module's code to use the locally installed copy of aws-sdk, not the one available in the lambda runtime.

[eavestn]

I can attest that I have seen the same issue but as it relates to deploying a layer for the AWS Lambda on a Node.js 14.x ARM env. No matter what I do in IAM I can't get the Lambda to discover the client-specific library:

const { S3Client } = require("@aws-sdk/client-s3");

const S3 = new S3Client({ region: process.env.AWS_REGION ?? "us-east-1" });

My IAM policy document (assinged via a role to the Lambda) is([xxxxx] below represents a real account no.):

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "",
            "Effect": "Allow",
            "Action": [
                "lambda:GetLayerVersion",
                "lambda:PublishVersion"
            ],
            "Resource": [
                "arn:aws:lambda:*:[xxxxx]:function:*",
                "arn:aws:lambda:us-east-1:[xxxxx]:layer:document-uploading-layer:2"
            ]
        },
        {
            "Sid": "",
            "Effect": "Allow",
            "Action": "lambda:PublishLayerVersion",
            "Resource": "arn:aws:lambda:us-east-1:[xxxxx]:layer:document-uploading-layer:2"
        },
        {
            "Sid": "",
            "Effect": "Allow",
            "Action": [
                "lambda:ListLayerVersions",
                "lambda:ListLayers"
            ],
            "Resource": "*"
        }
    ]
}

The only work around I have is to create a deployment package that includes the node_modules directory as a sibling of the index.js export.handler = file.

[eavestn]

I can attest that I have seen the same issue but as it relates to deploying a layer for the AWS Lambda on a Node.js 14.x ARM env. No matter what I do in IAM I can't get the Lambda to discover the client-specific library:

const { S3Client } = require("@aws-sdk/client-s3");

const S3 = new S3Client({ region: process.env.AWS_REGION ?? "us-east-1" });

My IAM policy document (assinged via a role to the Lambda) is([xxxxx] below represents a real account no.):

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "",
            "Effect": "Allow",
            "Action": [
                "lambda:GetLayerVersion",
                "lambda:PublishVersion"
            ],
            "Resource": [
                "arn:aws:lambda:*:[xxxxx]:function:*",
                "arn:aws:lambda:us-east-1:[xxxxx]:layer:document-uploading-layer:2"
            ]
        },
        {
            "Sid": "",
            "Effect": "Allow",
            "Action": "lambda:PublishLayerVersion",
            "Resource": "arn:aws:lambda:us-east-1:[xxxxx]:layer:document-uploading-layer:2"
        },
        {
            "Sid": "",
            "Effect": "Allow",
            "Action": [
                "lambda:ListLayerVersions",
                "lambda:ListLayers"
            ],
            "Resource": "*"
        }
    ]
}

The only work around I have is to create a deployment package that includes the node_modules directory as a sibling of the index.js export.handler = file.

This actually isn't true. I found this doc. for v. 14+ buried: https://docs.aws.amazon.com/lambda/latest/dg/configuration-layers.html#configuration-layers-path

The path has to match.

[FrancoCorleone]

Is it really that difficult to solve? I'm struggling with local development because of that...

[FSDevelop]

This is preventing people from using es6 features on Lambda which is not good

[timrobinson33]

as @EPMatt said, a simple workaround is just to move aws-sdk from your devDependencies into dependencies so it gets bundled with your package. Creates 70MB of extra bloat (per function obvs) but I'm sure that's a drop in the ocean to lambda/npm aficionados.

[FrancoCorleone]

This is by far the worst idea actually. That increases costs, makes longer to deploy etc. What I did to "solve" it for now was to install aws-sdk globally for my node version (handled by nvm) and then created a symlink from /var/runtime/*/aws-sdk to my globally installed sdk. At least I can work locally now. But this is a part of a bigger problem with using ESM in lambdas, look here for more information

[FSDevelop]

as @EPMatt said, a simple workaround is just to move aws-sdk from your devDependencies into dependencies so it gets bundled with your package. Creates 70MB of extra bloat (per function obvs) but I'm sure that's a drop in the ocean to lambda/npm aficionados.

I have this setup, not working neither. It's a problem of using the import keyword

[FSDevelop]

This is by far the worst idea actually. That increases costs, makes longer to deploy etc. What I did to "solve" it for now was to install aws-sdk globally for my node version (handled by nvm) and then created a symlink from /var/runtime/*/aws-sdk to my globally installed sdk. At least I can work locally now. But this is a part of a bigger problem with using ESM in lambdas, look here for more information

I don't think it could be a problem. It's slow when you zip everything and upload, but if you only upload your changes after initial upload or use sam || serverless it may skip the node_modules folder because it is already included.

[rmclaughlin-nelnet]

@vudh1 Any update on this? We are also running into this same problem. What is the official way to use the sdk that is built into the lambda runtime with es6?

[FrancoCorleone]

Seriously guys. Use my solution, problem solved until they finally fix it

[rmclaughlin-nelnet]

No offense, but your solution is a giant hack and as you stated in later post it makes local development difficult. I have read about symlinks being a possible solution to that, but it is one hack after another and fairly soon it gets so messy that maintaining our dev environment takes more work than programming our code.

For now it is easier to just go back to using CommonJs.

[FSDevelop]

For now it's easier to specify the full path. But not ideal. I noticed specifying full path is not necessary when using sam local invoke, but it is necessary when running up in aws.

[FrancoCorleone]

@rmclaughlin-nelnet No offense but that's BS :D But I get where maybe it gets confusing.So there are two issues:

1. Using layers with ESM approach - doesn't work. What I recommend is to create layer structure and simply, in package.json put that module as local dependency. Then during npm install, symlink is implicitly created and then you can zip it with library embedded.
2. Now, for the aws-sdk. Just install it "globally". For example in your root project in node_modules or in your nvm distribution. Whatever. And then create a symlink globally in your system that redirects from /var/runtime(...) to your local distribution. \

From that point on, you can simply work with it. Or you can stick with CommonJS, that works too

[kuhe]

The AWS SDK for JavaScript v2 is imported with the name aws-sdk (non-scoped). This is available in AWS Lambda, but only with require.

The AWS SDK JS v3, which is this repository, is not available in AWS Lambda currently, unless you upload it in your own application bundle. This one is imported with @aws-sdk/***-client and other packages under the @aws-sdk/ prefix. We are working with AWS Lambda to make the AWS SDK v3 available without needing to upload your own copy, but cannot provide a timeline.

[FSDevelop]

Thanks @kuhe !

[FrancoCorleone]

@kuhe but the truth is lambda is struggling with ES imports from layers altogether. So both have to be sorted for it to work.

[mcqj]

Thanks @kuhe The very first line of the V3 README says

The [version 3.x](https://github.com/aws/aws-sdk-js-v3) of the AWS SDK for JavaScript is generally available. For more information see the [Developer Guide](https://docs.aws.amazon.com/sdk-for-javascript/v3/developer-guide/) or [API Reference](https://docs.aws.amazon.com/AWSJavaScriptSDK/v3/latest/index.html).

Nowhere in the README are there any caveats about using it with Lambda. Perhaps, a note at the top of that file would save other developers a lot of time.