Exclude dev dependencies from Node.js runtime package

[alekbarszczewski]

Prerequisites

• [x] I am running the latest version. (up upgrade)
• [x] I searched to see if the issue already exists.
• [x] I inspected the verbose debug output with the -v, --verbose flag.
• [ ] Are you an Up Pro subscriber?

Description

I am getting Error: building: building: zip contents is 269 MB, exceeding Lambda's limit of 262 MB because my devDependencies in node_modules (Node.js project) are included in a zip file. I have quite a lot of them so putting them manually to .upignore is not convenient. Would be nice if there was an option to automatically exclude devDependencies from Lambda zip package. I know that serverless.js does this automatically when building Lambda package(s). For example here is a list of my dev dependencies:

"@babel/cli": "^7.0.0-beta.32",
    "@babel/core": "^7.0.0-beta.32",
    "@babel/node": "^7.0.0-beta.32",
    "@babel/plugin-proposal-class-properties": "^7.0.0-beta.32",
    "@babel/plugin-proposal-decorators": "^7.0.0-beta.32",
    "@babel/plugin-proposal-export-namespace": "^7.0.0-beta.32",
    "@babel/plugin-proposal-object-rest-spread": "^7.0.0-beta.32",
    "@babel/plugin-transform-classes": "^7.0.0-beta.32",
    "@babel/preset-env": "^7.0.0-beta.32",
    "@babel/preset-flow": "^7.0.0-beta.32",
    "@babel/register": "^7.0.0-beta.32",
    "babel-eslint": "^8.0.2",
    "babel-plugin-istanbul": "^4.1.5",
    "babel-plugin-root-import": "^5.1.0",
    "chai": "^4.1.2",
    "cross-env": "^5.1.1",
    "flow-bin": "^0.59.0",
    "inquirer": "^5.1.0",
    "mocha": "^4.0.1",
    "nock": "^9.1.3",
    "nyc": "^11.3.0",
    "opn-cli": "^3.1.0",
    "rimraf": "^2.6.2",
    "sinon": "^4.1.2",
    "standard": "^10.0.3",
    "up": "^1.0.1"

[lukeed]

I think you would have to do this manually in your build hook. You could do something like

{
  "build": "npm run build && rm -rf node_modules && npm install --only=production"
}

[tj]

It would be super cool to handle this magically, but it may be quite a bit of work, unless npm has something like npm list --production (will check).

Re-installing each time is definitely not ideal. Some people do it in CI so you're not constantly replacing node_modules, another alternative is bundling with browserify or similar—this is ideal for cold start performance as well.

[tj]

Hmm it does have prod listing, but node's boot time for running the npm CLI at all is pretty slow. I think I'd rather re-implement that part in Go, could be nice to add though!

   λ node-express (master):  npm ls --only=development --parseable | wc -l
     873
   λ node-express (master):  npm ls --parseable | wc -l
     905

It takes 2s just to run on a small app, but maybe that'll be fine as a stop-gap.

https://github.com/tj/node-prune helps too, honestly bundling with browserify is best if possible, since that strips all of the markdown and random files.

[alekbarszczewski]

Hmm so you advice to use browserify to make single file bundle and completely ignore node_modules?

[tj]

Yep, it has a few benefits: much smaller build since it ignores many files, much faster cold start times due to the syscall overhead of many thousands of require() calls, also gives you the ability to use whichever ES6 features you want if you use Webpack/Babel for example.

[alekbarszczewski]

Thanks I will try it. I never thought about browserifying server side code, always used pure babel to just transpile sepearate files, but in lambda environment it sounds reasonable.

[ghost]

I've seen some strange behaviour with ignoring dev dependencies. I have a next project - and the build process bundles into a dist. But if i ignore any of the source files (etc /pages) the deployed app fails. Same if i remove any node_modules - even dev dependencies. The build is client side - so Im wondering why this may have problems. If i run the build process manually - I can remove the source files and things run fine. [Perhaps is thread spamming - but seemed relevant to the discussion]

[tj]

@glenarama I can't comment on the Next stuff, I'm not sure what it expects, but that does sound strange. What's your package.json "start" script? (unless you have proxy.command defined in up.json), I wonder if maybe it's still configured to try building on-demand or something like that

[ghost]

Package.json Scripts: "scripts": { "dev": "next", "build": "next build", "start": "next start -p $PORT", }

up.json:

{ "name": "myapp", "profile": "myapp", "regions": ["eu-west-2"], "lambda": { "memory": 1536 }, "stages": { "production": { "domain": "app.myapp.global" }, "staging": { "domain": "appdev.myapp.global" } } }

When running up - it goes through the build process - obviously with a large number of files, the zip is very large. So im keen to be aggressive on my up.ignore filters.

[ghost]

My bad...you even warn about it in the docs:

Note that patterns are matched much like .gitignore, so if you have the following .upignore contents even node_modules/debug/src/index.js will be ignored since it contains src."

I was including "pages" rather than "./pages" which removed some of the dist files.

[tj]

ahh!! that'll do it, when in doubt you can do up -v to output all the files being added or filtered, or up build --size will list them by size

[timqian]

For now, the best solution I find is by using npm prune and npm i --offline command

# in the build hook, this command will remove dev dependencies
npm prune --production

# in the clean hook, bring the dev dependencies back without downloading anything
npm i --offline

As a example, you can find these configs in my project: https://github.com/t9tio/cloudquery/blob/master/up.json

Refs:

• npm prune: https://docs.npmjs.com/cli/prune.html
• npm i --offline: https://github.com/npm/npm/issues/2568#issuecomment-331753223

[timchambers]

The yarn equivalent of what @timqian proposed using prune in the up hooks looks like this, and seems to work well in my testing so far:

  "hooks": {
    "prebuild": "yarn install --production",
    "postdeploy": "yarn install --offline"
  },

[k00k]

The yarn equivalent of what @timqian proposed using prune in the up hooks looks like this, and seems to work well in my testing so far:

  "hooks": {
    "prebuild": "yarn install --production",
    "postdeploy": "yarn install --offline"
  },

We were successfully using @timchambers solution above until we tried to do this when NODE_ENV=production. When this is the case, yarn won't install devDependencies. To get past this you need to change your postdeploy (or clean) hook to "postdeploy": "yarn install --production=false". We are doing this on a per-stage basis. Our staging hooks still look like what @timchambers posted above, our change is only for production.

Here's some more info about how yarn works when NODE_ENV=production: https://github.com/yarnpkg/yarn/issues/2739

[timchambers]

We were successfully using @timchambers solution above until we tried to do this when NODE_ENV=production. When this is the case, yarn won't install devDependencies. To get past this you need to change your postdeploy (or clean) hook to "postdeploy": "yarn install --production=false". We are doing this on a per-stage basis. Our staging hooks still look like what @timchambers posted above, our change is only for production.

Here's some more info about how yarn works when NODE_ENV=production: yarnpkg/yarn#2739

@k00k yeah, that's a better approach. Here are my updated hooks. I switched to using the predeploy instead of prebuild hook since my latest project uses some devDependencies to build, so I need to keep those around until the build is finished, I also got rid of the --offline flag since Yarn's cache makes it plenty fast:

  "hooks": {
    "predeploy": "yarn install --production",
    "postdeploy": "yarn install --production=false"
  },

[pankuweb]

"hooks": { "predeploy": "yarn install --production", "postdeploy": "yarn install --production=false" },

In your package.json file

It's worked for me