build: generate multiple formats (cjs, esm)

[manju-reddys]

PR Checklist

Please check if your PR fulfills the following requirements:

• [x] The commit message follows our guidelines: https://github.com/nestjs/nest/blob/master/CONTRIBUTING.md
• [ ] Tests for the changes have been added (for bug fixes / features)
• [ ] Docs have been added / updated (for bug fixes / features)

PR Type

What kind of change does this PR introduce?

• [ ] Bugfix
• [ ] Feature
• [ ] Code style update (formatting, local variables)
• [ ] Refactoring (no functional changes, no api changes)
• [x] Build related changes
• [ ] CI related changes
• [ ] Other... Please describe:

What is the current behavior?

The current grunt build process can only generate cjs module format but the lots of other projects moved on to use the mjs (aks ESM) format.

Issue Number: N/A

What is the new behavior?

This PR adds new build process (prod) to generate multiple formats using tusp.

npm run build:multi will compile and generate cjs and mjs format and also *.d.ts

Does this PR introduce a breaking change?

• [ ] Yes
• [x] No

Other information

[manju-reddys]

Also, what was the reason for using tsup over the existing gulp process we have?

TSC can't generate the .mjs and the gulp compiles the TS using gulp-typescript which internally runs TSC.

[manju-reddys]

@jmcdo29 how do I build all the module's under the @nestjs scope?

[manju-reddys]

This PR is all good, I tested against my codebase (12+ apps) and everything seems to be working without any issue and not even typing issues I found.

[jmcdo29]

That's great. I'm still waiting to find out what problem ESM solves with regards to Node and Typescript. Seems it's brought about way more headaches than solutions

[JumpLink]

@jmcdo29 more and more node.js packages are switching to ESM. It can be very problematic to use this packages in a CJS module / app. E.g. I use node-fetch in my nest.js project. Since V3 I can't it use anymore so I need to stay on v2.x (async import has not worked for me). Which is a shame because I like to keep my project up to date.

And there are also other packages I can't upgrade for this reason.

See this gist, many packages wich are ESM only now are linking to that.

See also this blog post: https://blog.sindresorhus.com/get-ready-for-esm-aa53530b3f77

[jmcdo29]

@jmcdo29 more and more node.js packages are switching to ESM. It can be very problematic to use this packages in a CJS module. E.g. I use node-fetch in my nest.js project. Since V3 I can't it use anymore so I need to stay on v2.x. Which is a shame because I like to keep my project up to date.

And there are also other packages I can't upgrade for this reason.

See this gist, many packages wich are ESM only now are linking to that.

See also this blog post: blog.sindresorhus.com/get-ready-for-esm-aa53530b3f77

@JumpLink I've seen that more packages are using ESM, but my question still remains why? Is it just because it's now the accepted JS standard? I'm all for change and getting the ball rolling, but I want to know why everyone is switching to ESM, if there's something I'm missing other than "it's the new standard".

[JumpLink]

@jmcdo29 One reason is that you can also use such packages in the browser and I think it is also easier to use such packages in Deno. ESM is a cross-platform standard. CommonJS only for node.

There are also other JavaScript runtimes that cannot use CommonJS, for example GJS under GNOME.

The JavaScript world would be much easier and simpler if all would use the same module system.

Here are more reasons: https://twitter.com/sindresorhus/status/1349312503835054080?s=20

[jmcdo29]

@JumpLink so when it comes down to it, it's really because "this is the new standard" more than anything else, yeah?

The JavaScript world would be much easier and simpler if all would use the same module system.

This I agree with. I just don't know why ESM was necessary. It seemed that CJS was doing just fine as it was. Like I said before, I'm all for change, just want to know the why

[JumpLink]

@jmcdo29 You could also use browser modules in node easier. And other runtimes can use this modules, too. Node.js is the only one with another standard. So there are many runtimes wich are using the same module system and can be compatible with the others, node could also be compatible and all sides would benefit from that.

ESM also has fewer disadvantages. It is problematic to use ESM modules in CJS, but if your application uses ESM you can still integrate CJS modules in node without problems, the other way round not so well.

I want to use ESM modules in my Nest.js application but currently I can't.

By the way: I heard tree shaking should also be more manageable in ESM.

Summary:

• Top level await support
• Unified syntax
• Top-level await
• Import both default export and named exports in the same statement
• Re-export syntax
• Potentially faster import step
• Official loader hooks (still draft)
• Cross Runtime compatible
• Same syntax in TypeScript
• Potentially better tree shaking
• Less complexity, the tools no longer have to support multiple module systems
• No more converting / polifills between the module systems necessary (webpack, Typescript, Babbel, browserify, ...)

[jmcdo29]

You could also use browser modules in node easier.

Generally true until you start having the DOM types show up and Typescript starts throwing errors left and right because of environment differences.

ESM also has fewer disadvantages.

Not sure what disadvantages are being considered here other than use of ESM inside CJS or CJS inside ESM. To my knowledge mocking and many test frameworks still have some troubles with ESM. Not a huge deal in Nest projects due to being able to provide our own mocks and the dependency injection, but just something to call out.

By the way: I heard tree shaking should also be more manageable in ESM.

I'd be curious as to why this is. It's not like ESM uses a new AST, whatever is doable, in terms of tree shaking, in ESM I think should be doable in CJS as well.

I'll keep looking around and finding out more of why this change came in the first place and what kind of impact it will have overall.

[manju-reddys]

@JumpLink summarizes the benefits and one addition to it is the security which is the main concern in the crypto world as many node_modules are easy to introduce threats into CJS example.

const fs = require('fs'); 
// any method of the fs can be changed here after

in ESM

import {readFile} from 'fs';
// Not possible

[jmcdo29]

@JumpLink summarizes the benefits and one addition to it is the security which is the main concern in the crypto world as many node_modules are easy to introduce threats into CJS example.

const fs = require('fs'); 
// any method of the fs can be changed here after

in ESM

import {readFile} from 'fs';
// Not possible

Yep, this is why testing frameworks have a problem with figuring out how to do mocks for ESM, because you can't just change the original implementation like you can with CJS

[rat-matheson]

Angular 13 removed CommonJS outputs so libraries created with it only use ES modules. I have some custom angular libraries that I share with the backend and frontend and due to upgrading to angular 13, I can no longer build my nest backend as I get Error [ERR_REQUIRE_ESM]: require() of ES Module.

Will this commit correct that issue?

[wodka]

@jmcdo29 is there any way to help move this pr forward? There are more and more packages switching to ESM only and it would be great to be able to upgrade them...

like https://github.com/sindresorhus/serialize-error

as for why? https://gist.github.com/sindresorhus/a39789f98801d908bbc7ff3ecc99d99c

[kamilmysliwiec]

I personally share the concerns of everyone who collaborated (on looking for a viable solution and seamless path forward) in this issue https://github.com/fastify/fastify/issues/2847

[rat-matheson]

Any idea why this PR is blocked? Is there anything that can help move it along?

[victorm-sp]

Is possible to check this PR @jmcdo29 ? I'm not able to update my version of Angular to 13 and Angular 14 is almost here D:

[jmcdo29]

As Kamil has already mentioned, there's concerns around how ESM integrate with Node packages, as you can see in the linked Fastify discussion. I understand that Angular is using ESM, and that browser side JS ESM is the standard, and now Node can run in an ESM mode that supports ESM modules.

If you need to use an ESM module inside a Nest application, right now you can await import('esm-module') and make use of it that way (wrap it in a top level function if you need to). I'm still yet to see real reasons to change over to ESM other than it's what the browser uses and it's "the future" of JavaScript packages. Until I know why we'd want to change over to it, even though I see that there is a request to do so, I can't really feel comfortable with this.

[manju-reddys]

@jmcdo29 Why ESM is highlighted in the comments and i'm not understand what else is that you what to know.

@kamilmysliwiec @jmcdo29 The request is not change everything to ESM but publish both ESM + CJS which is very much possible with modern tooling ( as I already put together in this PR).

[jmcdo29]

I understand it's possible, but the question for us as maintainers is what kind of maintenance burden does this bring on us when something goes wrong on the ESM side? At least, that's my main concern when I don't know why we're trying to support ESM other than what my most recent comment mentioned

[manju-reddys]

Release it under beta/experimental, and as adoption or usage increase, the issues or burden to maintaining start coming in to evaluate whether it's worth the effort.

[kamilmysliwiec]

@manju-reddys and then if the burden becomes too high, deprecate it? This doesn't seem reasonable

[rat-matheson]

As an alternative option, would it be possible to update the documentation to include a method of converting ES modules to commonjs so that they can be used by nestjs (via babel or something similar)? Perhaps there is an easy configuration that can be used for angular cli generated libraries (for example). I haven't spent much time with babel so the burden for me to learn all the configuration/plugin/nuance is high but if there was a simple guide as one of the nest recipes, maybe it would mitigate some of the concerns in this thread.

The ES module issue is relevant in part because nestjs and angular use a similar design pattern. Angular developers are drawn to nestjs as it feels very natural. They are a good user base to help keep the project active. So perhaps there is some way to mitigate this issue so they can use it.

[victorm-sp]

Yeah i don't really try to combine Angular and NestJS packages, I'm only using it with another of your projects https://github.com/nestjs/ng-universal. I think if the support for ESM doesn't exist the ng-universal package will be outdated soon or later.

[wodka]

For me the main problem is that more packages stop shipping CJS. This makes them impossible to use in combination with NestJS.

it is just becoming more and more painful when packages go with only one of the two... With the ability of esm to also load cjs packages it should be at least for frameworks like nestjs possible to use esm. I know that there are problems with ESM in certain constellations. But the same is true without ESM.

to make it clear to everyone nest could write a startup message with a warning about potential esm problems (with the ability to deactivate it with an option like:

  const app = await NestFactory.create(AppModule, {
    hideEsmWarning: true,
  })

[blqke]

@jmcdo29 @kamilmysliwiec Do you guys now have a position on this subject ? This is not my domain of expertise, but I see more and more libraries maintainer are switching to ESM, deliberately hoping for the community to fully switch to ESM by rejecting CJS. Here is an example https://github.com/NotionX/react-notion-x/issues/277#issuecomment-1110254556, here an other one https://github.com/sindresorhus/meta/discussions/15

I know "following the trend" is never a good argument and that you really want to know why, but you may see more and more people (me included) getting blocked when they need to use an ESM package in NestJS. Could we make a weighted pros/cons list to see if it's worth it ?

[jmcdo29]

@jmcdo29 @kamilmysliwiec Do you guys now have a position on this subject ? This is not my domain of expertise, but I see more and more libraries maintainer are switching to ESM, deliberately hoping for the community to fully switch to ESM by rejecting CJS. Here is an example NotionX/react-notion-x#277 (comment), here an other one sindresorhus/meta#15

I know "following the trend" is never a good argument and that you really want to know why, but you may see more and more people (me included) getting blocked when they need to use an ESM package in NestJS. Could we make a weighted pros/cons list to see if it's worth it ?

I think with Typescript 4.8 having stronger full support for ESM might make this a more feasible thing to manage for us rather than trying to support CJS and ESM in parallel. Currently, I haven't come into any situation where ESM has blocked me, but I'm probably not using the same packages as everyone else. I've been looking for what the reason for the move is, and every time I find a list all I can think is "we can already do this in CJS" so it's hard for me to be gung ho about the migration. I'll communicate with Kamil and see if he agrees on the TS 4.8 idea. This will almost definitely have to come as part of a major package publish unless we manage the dual package types, which I don't think would be a great approach

[rat-matheson]

Thanks for considering it. We understand that maintenance is an issue and certainly appreciate all the hard work that has been done to make nestjs the wonderful tool that it is.

Not being able to import .mjs packages seems like a pretty big issue to me given that many libraries are shipping mjs only. Having some clear documentation on transpiling these packages down to commonjs so that we can import than into our nest applications would certainly help mitigate the issue.

[jmcdo29]

Can't you still import mjs file though? Just using the await import() method instead of an import at the top of the file?

[rat-matheson]

At the time I was having the issue, I can't remember if I tried that or not so perhaps that was all I needed to do. I know I never was able to resolve the issue I was having and downgraded packages instead. I've since moved on but when I try again (which I will at some point in the future), I'll update this thread with what happens.

[kamilmysliwiec]

Not being able to import .mjs packages seems like a pretty big issue to me given that many libraries are shipping mjs only. Having some clear documentation on transpiling these packages down to commonjs so that we can import than into our nest applications would certainly help mitigate the issue.

Can you point us at crucial packages (consisting of more than 1 file) that are currently only shipping mjs now? @rat-matheson

[rat-matheson]

For me, it was angular 13 where I first noticed it (release notes). The dropped support for commonjs and only build .mjs format. I share some logic & models between my backend and front end and I was no longer able to do that with this new angular release since I could only build mjs packages. As a result, I downgraded back to 12 and it seems like I'm stuck there for the foreseeable future.

Since this issue was raised, I've come across other packages that are mjs only as well but none of them have been critical to my dev process so far. It does seem to be the way things are moving.

[kamilmysliwiec]

I'd be OK with a hybrid approach that would let us provide both ESM and CJS at the same time as long as it doesn't impose a heavy maintenance burden. As for fully migrating to ESM (and dropping CJS support), I'd say we'll likely wait for other tools we integrate with before we migrate (drivers like fastify, express, @nestjs/microservices transporters, and test runners like jest).

[yf-hk]

For me, I don't really care which way Nest.js uses internally, but at least there should be a decent solution to import third party ESM modules. A lot of modules dropped support for cjs mode. For example p-map, lodash-es. It's very weird to load like p-map using pMap = await import('p-map');. It loses static checking for TypeScript as well.

[manju-reddys]

Can you point us at crucial packages (consisting of more than 1 file) that are currently only shipping mjs now? @rat-matheson

https://www.npmjs.com/package/node-fetch

[yf-hk]

Can you point us at crucial packages (consisting of more than 1 file) that are currently only shipping mjs now? @rat-matheson

execa (55,978,898 weekly downloads) p-map (35,969,401 weekly downloads), p-all, p-filter, p-times, p-retry, p-queue etc. and dozes or even hunderds more..

[yf-hk]

For now as a workaround, is it possible to add https://babeljs.io/docs/en/babel-plugin-transform-modules-commonjs?

[muratbeser]

Not being able to import .mjs packages seems like a pretty big issue to me given that many libraries are shipping mjs only. Having some clear documentation on transpiling these packages down to commonjs so that we can import than into our nest applications would certainly help mitigate the issue.

Can you point us at crucial packages (consisting of more than 1 file) that are currently only shipping mjs now? @rat-matheson

Hi, since 98 I been developing applications libraries, over time I saw many frameworks to come and go. I really like stable approach of NestJS I would like to put my two cents.

Lately I started to create an NestJs application uses nrwl/nx ... @prisma/prisma and Angular and with that as you can imagine some type definitions are started to pop-up. so I though what I could do is lets create a proxy for prisma for angular, then validate the query with JSON Schema, so far everything is fine. Since I could use the services that I wrote inside NestJS I said lets create a private npm package so I can use code I wrote both sides.
To solve this issue I needed write a custom package builder that uses nrwl's build system. So node can understand when to use which code base as you can imagine I went into internals of nrwl's code base just to so I can gain time in future projects and such.

As developers we don't want to maintain so many different builds/code base. If I can share encapsulate a code tool and use it in many env. I would chose to do it. as I see it JS(not browser or node) community making a choice here and they are using their power to enforce ESM modules. I do strongly suggest at least making a research before it's too late to catch up.

Also I think ESM support on NestJS will also push other people to think on migrating to ESM.

and I'm sorry if my English has problems not my main language.

[Ghirigoro]

@jmcdo29

If you need to use an ESM module inside a Nest application, right now you can await import('esm-module') and make use of it that way (wrap it in a top level function if you need to).

You can't use top-level await in TypeScript in a CommonJS project, so you have to wrap the imports in an async function. Given how async works, that means that anything that uses an ESM module (even transitively) has to be async. That's fine if you have a single isolated dependency but if you have more than a couple of ESM-only modules you find yourself having to make just about EVERY function in your project async. Also this will not work for constructors since they can't be async, so you can't use ESM modules during initialization which is a major problem with this approach.

Alternatively you can inject the dependencies into the constructor of your service/controller/etc. (see here ) but that's hardly a minor tweak.

The point of all this is that there's not really any trivial workaround to importing ESM modules with TypeScript (at least right now).

I'm still yet to see real reasons to change over to ESM other than it's what the browser uses and it's "the future" of JavaScript packages.

Unfortunately, I think the only reason you're ever going to get for this is that "it's what the browser uses". I mean, no one in their right mind would have chosen JavaScript to be a backend programming language in 2009, given its lack of a module system, etc. but here we are because "it's what the browser uses" ;)

We all really do appreciate the huge effort that you and everyone else maintaining Nest is making, and have no doubt that transitioning from CommonJS is a big undertaking, but at this point it seems unavoidable, so the question becomes when will that transition be necessary. Only you and the rest of the maintainers can make that determination but I (and the rest of your user-base) can offer you some data to indicate how important (or not) ESM support is.

For me at least, ESM support is pretty important (8 out of 10). I work on a monorepo that is designed to have a lot of code shared between the front and the back ends. Even outside of ESM-only npm packages, there's increasing pressure coming from front end tools like Vite and Vitest that are built around ESM. Much of it I can work around, but it's increasingly hard to justify when Nest is the only part of our stack that requires CommonJS.

Do with that what you will, and thanks for all your efforts.

[rubiin]

Having issues on my case as well with packages like graphql-upload and nanoid. Nanoid is not that important tbh but grapql-upload has recently moved to ESM and i cannot seem to use the latest version(with some major patches) due to this cc. https://github.com/jaydenseric/graphql-upload/issues/329

[tfohlmeister]

Having issues as well with packages like @angular/localize. Workaround did not work for me, as the package itself imports modules in an unsupported way it seems.

Would really like to see ESM support in nestjs!

[rat-matheson]

@rubiin

Having issues on my case as well with packages like graphql-upload and nanoid. Nanoid is not that important tbh but grapql-upload has recently moved to ESM and i cannot seem to use the latest version(with some major patches) due to this cc. jaydenseric/graphql-upload#329

I got nanoid working, but I use version ^3.3.4 and not the latest version 4.0.0. It is going to be a problem to stay on nestjs if I can no longer update supporting libraries

[rubiin]

@rubiin

Having issues on my case as well with packages like graphql-upload and nanoid. Nanoid is not that important tbh but grapql-upload has recently moved to ESM and i cannot seem to use the latest version(with some major patches) due to this cc. jaydenseric/graphql-upload#329

I got nanoid working, but I use version ^3.3.4 and not the latest version 4.0.0. It is going to be a problem to stay on nestjs if I can no longer update supporting libraries

Well you can still use the older versions but you will be missing out new features as well as patches that are using ESM now. Problem occurs when there are zero day exploits and even if there are patches realeased, you cant use them as the project has migrated to ESM

[nanoTitan]

I'm having problems trying to use this method with ipfs-http-client: const { create } = await import('ipfs-http-client') isn't working for me using an async function. The result is always undefined. Any advice or ideas are appreciated.

[nanoTitan]

I found something that works, but it is a hack based on Jay Wolfe's post on commonjs to ESM conversion: https://jaywolfe.dev/how-to-use-es-modules-with-older-node-js-projects-the-right-way/

Define a new dynamic import that prevents typescript from rewriting your import as a require: const dynamicImport = async (packageName: string) => new Function(return import('${packageName}'))();

Then, you can call your import asynchronously const { create } = await dynamicImport('ipfs-http-client');

Give this a try if normal dynamic imports aren't working and you are using typescript.

[conioX]

@rubiin

Having issues on my case as well with packages like graphql-upload and nanoid. Nanoid is not that important tbh but grapql-upload has recently moved to ESM and i cannot seem to use the latest version(with some major patches) due to this cc. jaydenseric/graphql-upload#329

I got nanoid working, but I use version ^3.3.4 and not the latest version 4.0.0. It is going to be a problem to stay on nestjs if I can no longer update supporting libraries

Well you can still use the older versions but you will be missing out new features as well as patches that are using ESM now. Problem occurs when there are zero day exploits and even if there are patches realeased, you cant use them as the project has migrated to ESM

got also move to esm :(

[kbzowski]

Lazy solution is just wrap each ESM lib into service which use https://github.com/cspotcode/tsimportlib or different solution mentioned in this thread. In case of https://github.com/sindresorhus/execa v6 that service can look like:

import {Injectable} from "@nestjs/common";
import {dynamicImport} from "tsimportlib";

export type ExecaModule = typeof import('execa')
export type Execa = typeof import('execa').execa;
export type ExecaCommand = typeof import('execa').execaCommand

@Injectable()
export class ExecaService {
    private execaModule : ExecaModule;

    public execa: Execa;
    public execaCommand: ExecaCommand;
    
    constructor() {
        this.loadExeca().then(() => {}).catch(err => {
            // Suitable error handling
        });
    }

    async loadExeca() {
        this.execaModule = (await dynamicImport(
            'execa',
            module,
        )) as ExecaModule;
        this.execa = this.execaModule.execa;
        this.execaCommand = this.execaModule.execaCommand;
    }
}

It is known that such a solution does not solve all use cases, but for me it works without a problem. Of course, it would be good to abandon them at some point and have support for ESM in NestJS.