6MB AWS Lambda Hard Limit Strikes Again

[jonsharratt]

This is a Bug Report

Description

For bug reports:

• What went wrong? When attempting to use npm i npm --save failed.

It actually came about attempting to install cordova as npm is a dependency of it.

• What did you expect should have happened? npm module should have installed

• What stacktrace or error message did you experience?

Similar or dependent issues:

• #39

Additional Data

• NPM CLI version you are using: 3.10.16
• Serverless version you're using: 1.6.1
• Node version you're using: v6.2.0

[jonsharratt]

CloudWatch Logs:

Lambda:

API Gateway:

[brimworks]

BTW, I also ran into this issue... wasted a few hours trying to figure out what went wrong. To bad we can't send an S3 signed PUT URL down to the client. Perhaps if we started a conversation with the npm developers we could change the client/server interaction during publish such that an S3 signed URL could be used?

[jonsharratt]

Yer @brimworks we really are shackled by the way npm and Lambda works just now. I spoke to Amazon and they said, they can not do anything for me in regards to these hard limits.

I have some things in the pipeline just now that will probably make the whole js community go crazy :trollface: with a codebox package manager that uses streams against a codebox private registry.

One thing I am aiming to achieve also is that each module is it's own ".json" file so that when we work with Docker it can cache every layer and only have to invalidate those dependencies that have a version change.

Watch this space 👍

[joebowbeer]

I think it would be worth mentioning this issue in the README.

It may also be worth mentioning that one strategy for escaping the 6MB limit is to configure codebox-npm to manage a single (private) scope, in which case the default registry is used for everything else:

npm config set @myscope:registry https://xyz.amazonaws.com/stage/registry/

"publishConfig": {
  "@myscope:registry": "https://xyz.amazonaws.com/stage/registry/"
},

[micksabox]

I’m having trouble parsing this issue and what it means for my decision to deploy Codebox-npm for my organization.

Does the 6mb limit mean that packages published or used via npm install cannot be over 6mb? I think this is why you described an error when attempting to install npm. I think this is also why @joebowbeer solution works for publishing/using npm modules from the public registry. But it still would not work if the scoped private modules go over 6mb in size, correct?

Thanks for the clarification and great work

[joebowbeer]

@micksabox AWS Lambda limits response body payload (e.g., your npm package) to 6MB

https://docs.aws.amazon.com/lambda/latest/dg/limits.html

[JordanSinko]

I was able to bypass this, specifically for the package 'npm', by forking the code and changing the response of the package-get be a redirect, instead of directly fetching its meta, when fetching NPM packages.