ERR_REQUIRE_ESM when used as a dependency with Webpack

[hubertgrzeskowiak]

I am building an app using Nx for modularisation. With the default Webpack 5 config for Node I get an error:

/Users/hgrzeskowiak/hugo-game/fresh-nx-geckos/nx-geckos-test/dist/apps/geckos-server-app/main.js:8
module.exports = require("@geckos.io/server");
                 ^
Error [ERR_REQUIRE_ESM]: require() of ES Module /Users/hgrzeskowiak/hugo-game/fresh-nx-geckos/nx-geckos-test/node_modules/@geckos.io/server/lib/index.js from /Users/hgrzeskowiak/hugo-game/fresh-nx-geckos/nx-geckos-test/dist/apps/geckos-server-app/main.js not supported.
Instead change the require of index.js in /Users/hgrzeskowiak/hugo-game/fresh-nx-geckos/nx-geckos-test/dist/apps/geckos-server-app/main.js to a dynamic import() which is available in all CommonJS modules.
(...)

Here are steps to reproduce:

$ npx create-nx-workspace@latest nx-geckos-test
Need to install the following packages:
  create-nx-workspace@latest
Ok to proceed? (y) y
✔ What to create in the new workspace · express
✔ Application name                    · geckos-server-app
✔ Use Nx Cloud? (It's free and doesn't require registration.) · No

>  NX  Nx is creating your v13.4.1 workspace.

  To make sure the command works reliably in all environments, and that the preset is applied correctly,
  Nx will run "npm install" several times. Please wait.

✔ Installing dependencies with npm
✔ Nx has successfully created the workspace.

$ npm run start

> [email protected] start
> nx serve

> nx run geckos-server-app:serve
chunk (runtime: main) main.js (main) 552 bytes [entry] [rendered]
webpack compiled successfully (26ac6f288b082854)
Debugger listening on ws://localhost:9229/70a1d9b4-af2b-4fc9-90f7-f3502e5fdf2e
Debugger listening on ws://localhost:9229/70a1d9b4-af2b-4fc9-90f7-f3502e5fdf2e
For help, see: https://nodejs.org/en/docs/inspector
Issues checking in progress...
Listening at http://localhost:3333/api

At this stage, the default Express app builds and runs.

Now I replace the contents of apps/geckos-server-app/src/main.ts to the Geckos server example from the README...

...and install the geckos server using

$ npm install @geckos.io/server

added 4 packages, and audited 785 packages in 7s

87 packages are looking for funding
  run `npm fund` for details

found 0 vulnerabilities

Ok, time to run it again:

❯ npm run start

> [email protected] start
> nx serve


> nx run geckos-server-app:serve
chunk (runtime: main) main.js (main) 610 bytes [entry] [rendered]
webpack compiled successfully (90920432a9d246ea)
Debugger listening on ws://localhost:9229/dc6d9e1a-1a15-432f-a46c-d2ac83b923ea
Debugger listening on ws://localhost:9229/dc6d9e1a-1a15-432f-a46c-d2ac83b923ea
For help, see: https://nodejs.org/en/docs/inspector

/Users/hgrzeskowiak/hugo-game/fresh-nx-geckos/nx-geckos-test/dist/apps/geckos-server-app/main.js:8
module.exports = require("@geckos.io/server");
                 ^
Error [ERR_REQUIRE_ESM]: require() of ES Module /Users/hgrzeskowiak/hugo-game/fresh-nx-geckos/nx-geckos-test/node_modules/@geckos.io/server/lib/index.js from /Users/hgrzeskowiak/hugo-game/fresh-nx-geckos/nx-geckos-test/dist/apps/geckos-server-app/main.js not supported.
Instead change the require of index.js in /Users/hgrzeskowiak/hugo-game/fresh-nx-geckos/nx-geckos-test/dist/apps/geckos-server-app/main.js to a dynamic import() which is available in all CommonJS modules.
    at [email protected]/server (/Users/hgrzeskowiak/hugo-game/fresh-nx-geckos/nx-geckos-test/dist/apps/geckos-server-app/main.js:8:18)
    at __webpack_require__ (/Users/hgrzeskowiak/hugo-game/fresh-nx-geckos/nx-geckos-test/dist/apps/geckos-server-app/main.js:32:41)
    at /Users/hgrzeskowiak/hugo-game/fresh-nx-geckos/nx-geckos-test/dist/apps/geckos-server-app/main.js:45:18
    at /Users/hgrzeskowiak/hugo-game/fresh-nx-geckos/nx-geckos-test/dist/apps/geckos-server-app/main.js:59:3
    at Object.<anonymous> (/Users/hgrzeskowiak/hugo-game/fresh-nx-geckos/nx-geckos-test/dist/apps/geckos-server-app/main.js:64:12)

I tested quite a few Webpack config overrides and using type: "module" in package.json, but to no avail. The app works when I manually change the require to import in the transpiled output, but that's not a sustainable solution.

I suspect the error has something to do with Geckos being an ESM package, and Webpack only using require instead of import as it should, but I couldn't find any way of changing that behaviour.

Any ideas?

Node version 16.13.0 Nx version 13.4.1 Webpack (transitive dep of Nx): 5.65.0

[yandeu]

Version 2 is ESM only. Maybe this helps: https://github.com/geckosio/geckos.io/issues/114

[hubertgrzeskowiak]

Cheers for the quick reply @yandeu !

The example repo you linked in the other ticket uses a nightly build of TypeScript. Looking at the release docs of TS 4.5:

Experimental Nightly-Only ECMAScript Module Support in Node.js (...) In turn, this feature is still available for use, but only under nightly releases of TypeScript, and not in TypeScript 4.5.

This explains why it wouldn't work for me and some other people. We are all using the latest, but stable releases, not nightly builds.

At this point I am not sure whether it's easier to automatically search and replace require("@geckos.io/server") with import("@geckos.io/server") and let NodeJS 16 handle it using its experimental ESM support, or whether I should look at forking Geckos as a CommonJS module.

[yandeu]

ESM is stable in nodejs 12, 14, 16 and above.

[yandeu]

whether I should look at forking Geckos as a CommonJS module

You should not. There are more and more ESM-only packages on npm. It doesn't make sense forking them.

[hubertgrzeskowiak]

Here's the cleanest solution I could find that works with TS 4.5:

A custom Webpack plugin that changes the type of the external module to import.

// TypeScript before version 4.6 does not support transpiling ESM imports to
// `import()`, but uses `require()` instead. NodeJS does not support the use
// of `require() for ECMAScript modules (ESM).
//
// Good reads:
// https://www.typescriptlang.org/docs/handbook/esm-node.html
// https://devblogs.microsoft.com/typescript/announcing-typescript-4-5/#esm-nodejs

/**
 * A Webpack 5 plugin that can be passed a list of packages that are of type
 * ESM. The typescript compiler will then be instructed to use the `import`
 * external type.
 */
class ESMLoader {
  static defaultOptions = {
    esmPackages: "all"
  };

  constructor(options = {}) {
    this.options = { ...ESMLoader.defaultOptions, ...options };
  }

  apply(compiler) {
    compiler.hooks.compilation.tap(
      "ECMAScript Module (ESM) Loader. Turns require() into import()",
      (
        compilation
      ) => {
        compilation.hooks.buildModule.tap("ECMAScript Module (ESM) Loader. Turns require() into import()", (module) => {
            if (
              module.type === "javascript/dynamic" &&
              (
                this.options.esmPackages === "all" ||
                this.options.esmPackages.includes(module.request)
              )
            ) {
              // All types documented at
              // https://webpack.js.org/configuration/externals/#externalstype
              console.log(module.request);
              module.externalType = "import";
            }
          }
        )
        ;
      }
    );
  }
}

module.exports = function(webpackConfig, context) {
  // NodeJS supports dynamic imports since version 12.
  webpackConfig.target = "node16";
  webpackConfig.plugins.push(
    new ESMLoader({esmPackages: "@geckos.io/server"})
  );
  return webpackConfig;
};

The custom webpack config can be set on an Nx app through the project.json:

{
  "targets": {
    "build": {
      "executor": "@nrwl/node:build",
      "options": {
       ...
        "webpackConfig": "apps/your-app/webpackConfig.js"
      },
      ...

This solution also requires tsconfig.json to have set the module type to something more modern than commonjs, e.g. es2020:

{
  "extends": "./tsconfig.json",
  "compilerOptions": {
    "outDir": "../../dist/out-tsc",
    "module": "es2020",
    "types": ["node"],
  },
  "exclude": ["**/*.spec.ts", "**/*.test.ts"],
  "include": ["**/*.ts"]
}

Once this is done, the built file can be ran using the usual nx serve.

[yandeu]

Sounds good.

Before ts v4.6 I also use es2020. With ts v4.6 I use node12 or nodenext.

I personally prefer keeping ts v4.6 as it will be release soon (next year).

[x89-h]

Just my opinion:

The purposeful restriction to use import is stupid. Import offers little to no benefit over require and require matches JS syntax much better than import. However, I can see how import is nice for people coming from other languages to JS.

// What variable declaration syntax is
const variable = value;

// What `require` syntax is
// function(parameter) outputs value;
const variable = function(parameter);

// What `import` syntax is
// `from parameter` outputs value
import variable from parameter;

As you can see import doesn't match JS variable declaration syntax as well. I don't fault the creators of geckos for this. It was someone's decision a while back to make a more efficient require and instead of keeping old syntax they decided to make something new. And these days the efficiency difference isn't even that great.

I am not a fan of "CommonJS"'s module.exports = ...;. I would prefer it to be export(...) or even module.export(...);

I am able to require in other ESM packages. The only one that doesn't work is geckos. Annoying because then I have to change all of my requires to imports just to get there to be some kind of uniform syntax.

import also all but destroys the possibility of doing something like this:

const object = {
    key1: require(someModule)
};

// How one would do it with `import`
import key1 from parameter;
const object = {
    key1
}

Just me ranting. No need to really pay attention to this.

[Ciberusps]

i fixed issue with ESM by using importType "module" in nodeExternals

[github-actions[bot]]

This issue is stale because it has been open 300 days with no activity. Remove stale label or comment or this will be closed in 10 days.

[github-actions[bot]]

This issue was closed because it has been stalled for 10 days with no activity.