Issue with duplicate/nested node_module versions of external dependencies

[oxc]

Context

We're using esbuild to bundle function code that we deploy to AWS Lambda. We have some rather large dependencies that blow up the function bundles too much, so we have put them into a lambda layer (in a node_modules directory) and mark them external in esbuild.

This works as long as there is only a single version of each dependency in the tree. As soon as one of the dependencies (explicit or transitive) requires a module spec that resolves to a different version than the top-level version, this gets deployed to a nested node_modules folder.

Example

For example, consider the following minimal project:

// package.json

{
  "name": "esbuild-external-issue",
  "version": "1.0.0",
  "main": "index.js",
  "license": "MIT",
  "dependencies": {
    "request": "^2.88.2",
    "uuid": "^9.0.0"
  }
}

// index.js

const { v4 } = require("uuid");
const request = require("request");

console.log("Hello World.");

If I install this (for example with yarn install), this will lead to the following folder structure (excerpt):

.
├── index.js
├── node_modules
│   ├── request
│   │   ├── node_modules
│   │   │   └── uuid // version 3.4.0
│   ├── uuid // version 9.0.0

What works

Now if I bundle this with esbuild, it correctly bundles both versions of uuid into the output:

$ npx esbuild --bundle --platform=node --outfile=out.js index.js

  out.js  1.3mb ⚠️

⚡ Done in 31ms

$ node out.js
Hello World.

Same if I mark all packages as external (via --packages=external or manually via --external:request --external:uuid):

$ npx esbuild --bundle --platform=node --outfile=out.js index.js --packages=external

  out.js  105b

⚡ Done in 1ms

$ node out.js
Hello World.

What doesn't work

However, if I only mark uuid as external, things break:

$ npx esbuild --bundle --platform=node --outfile=out.js index.js --external:uuid

  out.js  1.3mb ⚠️

⚡ Done in 30ms

$ node out.js
node:internal/modules/cjs/loader:571
      throw e;
      ^

Error [ERR_PACKAGE_PATH_NOT_EXPORTED]: Package subpath './v4' is not defined by "exports" in /home/obeliks/workspace/esbuild-external-issue/node_modules/uuid/package.json
    at new NodeError (node:internal/errors:399:5)
    at exportsNotFound (node:internal/modules/esm/resolve:361:10)
    at packageExportsResolve (node:internal/modules/esm/resolve:697:9)
    at resolveExports (node:internal/modules/cjs/loader:565:36)
    at Module._findPath (node:internal/modules/cjs/loader:634:31)
    at Module._resolveFilename (node:internal/modules/cjs/loader:1061:27)
    at Module._load (node:internal/modules/cjs/loader:920:27)
    at Module.require (node:internal/modules/cjs/loader:1141:19)
    at require (node:internal/modules/cjs/helpers:110:18)
    at node_modules/request/lib/auth.js (/home/obeliks/workspace/esbuild-external-issue/out.js:43457:16) {
  code: 'ERR_PACKAGE_PATH_NOT_EXPORTED'
}

Node.js v18.16.0

The reason for this failure is the following: any code that imports uuid (or in this case uuid/v4) within request code gets bundled to require('uuid/v4'), and now actually imports the uuid module from node_modules/uuid, not node_modules/request/node_modules/uuid. Since these versions are not compatible, this breaks.

What I tried

I tried to use --external:./node_modules/uuid/*, which sort of works, but that creates a call to require("./node_modules/uuid/dist/esm-node/index.js");, which will not work equivalently to require("uuid");, and does not resolve correctly if called from a deeper path (require("../../../node_modules/uuid/dist/esm-node/index.js");).

I also tried a custom onResolve plugin, which only marks imports that resolve to a top-level node_modules folder as external. This works, but is a bit cumbersome (especially to integrate from AWS CDK, but that's not really esbuild's fault).

Conclusion

While the custom plugin works, this seems like a very common problem. In fact, everybody who uses external for just a subset of packages is susceptible to this issue.

It would be nice to have an out-of-the-box way to handle this.

If there was a way to make --external:./node_modules/uuid/* keep the original import, that would probably be sufficient.

[evanw]

Isn't that what --packages=external is for? Why not use that? Alternatively, why aren't you doing --external:request if request is the problematic package?

It sounds like this problem has to do with how node works, and isn't really something specific to esbuild. You're trying to generate a single file where the import path uuid resolves to two different things but that's not how node's import system works.

[oxc]

This is not about one package being problematic. This happens whenever there are multiple versions of one package in the module tree that are not (and especially cannot be) deduped. A solution where I have to check and possibly change something whenever the dependency tree changes is too brittle.

There are also several reasons why I don't want to externalize all packages, foremost that I want to leverage the optimizations esbuild has to offer, tree-shaking etc. That's why --packages=external is also not a viable solution.

You're right, this happens because I want to make one file out of what were previously many, but that is what esbuild's bundle command is for, right? So I wouldn't say this is not specifically related to esbuild.

In fact I'm trying to generate a single file where multiple imports of 'uuid' don't point to different things. I want one import to be unchanged and point to 'uuid', and the others to be bundled.

(Rewriting identical imports to point to multiple different external modules is of course possible, but I agree that this is very node and deployment specific, and can thus well be done in a plugin. I still think this use case is more generic.) Basically I would like to keep some imports in their original form when they would resolve to a certain path, which is actually what I thought --external:./path/* would do.

[oxc]

If you think this is too specific for esbuild, I can luckily make it work with a plugin. However I believe that this is in fact mostly people actually want/expect when they specify a single module as --external, at least for anything that runs on node.

I would even say that the documentation is a bit misleading there:

If you want to customize which of your dependencies are external and which ones aren't, then you should be using external instead of this setting.

This suggests that --external takes a module as parameter, when in fact it takes an import, which does not care what it resolves to.

Of course in many (smaller) use cases this issue does not come to light, plus when it does, it is very hard to figure out the nature of the problem. In fact we faced the same problem a few months ago and back then I manually deduped the modules by upgrading some dependencies. Why exactly nested node_modules seemed to be ignored by esbuild, I could not fully understand at that time. It now makes sense of course, but I'm not surprised that I didn't find another report for this problem.

EDIT: What's more, this problem only becomes apparent when there is a major incompatibility between two versions of a package (like the failing import from uuid/v4). Only then will users even notice that there is something wrong. There might be subtle differences only in minor versions, leading to strange bugs, but discovering that they come from an unexpected version of a library is not trivial.

[hugomallet]

Hi, any update on this issue ?