Does the interop story actually work?

[zenparsing]

Let's assume that I'm creating a new package and I want to:

• Develop and publish without a transpiler, as ESM modules.
• Allow transpiler users to depend upon my package.

If the consumer is using a transpiler, then import declarations will be transformed into require calls. When the consumer requires my package they will get an error because I don't have an "index.js" or a CJS main field.

To fix this, I create a CJS entry point into my package (say an "index.js" file).

Inside of that "index.js" file, I want to pull in my package code and expose it using module.exports, so I try this:

module.exports = import("./default.js");

But import() returns a Promise, so this actually doesn't work.

There doesn't appear to be a way for me to pull in my module code and expose that module to require.

So my only option is to publish both CJS and ESM versions of each module in my package. I need to use a transpiler. But now I have two copies of my codebase to support, which is bad. I might just as well only publish the CJS version for consumers to use.

But now I'm right back to the current situation where I code in ESM and then transpile down to CJS.

It appears that as long as there is no way to synchronously load ESM modules from CJS modules, we are stuck and the ecosystem cannot move forward.

This seems pretty bad for the ecosystem.

Thoughts?

[guybedford]

If I publish a package as just ESM, transpiler users can use it with the "module" field as they already do through build tools like Parcel, Rollup and Webpack which already support this. Getting these tools to support "default.js" would be a good step as well if this spec can get some momentum.

I'm not sure "blanket transpiling of node_modules package folders" is a good strategy for transpiler users otherwise though.

Rather, I think one would ideally need to run transpilation on publish to offer "two versions" of a package. Tools could easily provide this service for users as a publish hook or similar in npm.

The basic workflow would then output a index.js transpiled from the default.js creating a "dual-mode package".

[guybedford]

Also tools like esm can support the "module" field loading in legacy non-module NodeJS.

[zenparsing]

Rather, I think one would ideally need to run transpilation on publish to offer "two versions" of a package. Tools could easily provide this service for users as a publish hook or similar in npm.

Right, so the CJS to ESM transition story becomes something like "everyone publishes two versions of everything" during the transition period. It's certainly not ideal but it might not be too terrible for the ecosystem.

But does this still work if we allow top-level await? If I want "dual mode" packages, then it seems like I can't use top-level await. Is that right?

[guybedford]

Yes, top-level await is a language feature that can't be transpiled (kinda like proxies really).

[zenparsing]

So we can say that synchronous loading and evaluation of modules is necessary for interop. To put it another way, any ESM module that depends on async is not interoperable.

For the sake of argument, let's see if there's a way to get around the requirement that a library author has to transpile-on-publish.

Let's call the set of all ESM modules E. There is another set Ei which is the subset of E that is interoperable. That is, the members of Ei can be loaded and evaluated synchronously. A module graph that only contains members of Ei is an interoperable module graph.

From a specification point of view, there is no particular reason that a module cannot be loaded and evaluated synchronously. Is there any reason why Node cannot provide a "synchronous" import capability for interoperable module graphs?

Specifically, I'm thinking of something like require.importSync(specifier) which would throw an error if any module in the resulting graph is not a member of Ei. This would help us to avoid the situation where I can't drop the transpiler until all of my dependents drop their transpilers.

[guybedford]

@zenparsing as far as I'm aware there aren't any implementation hurdles against synchronous ES module instantiation in the v8 API, but for Node:

1. The entire module pipeline is built asynchronously - having a synchronous pipeline would need to be a whole new pipeline - duplicating all the loader code and causing some maintenance overhead
2. The loader hooks are all designed to be async. Allowing synchronous loading of ES modules means loader hooks would need to either be sync or disabled. This is quite hard to manage from an API expectations point of view.

Nothing impossible... but it's just hard work and very unwieldy and bloat on top of what is there already.

There may be other arguments here I'm not aware of too.

[guybedford]

Personally though I see this as something that userland can provide - as mentioned previously using a project like @jdalton's esm to offer this support sounds ideal to me.

[zenparsing]

OK, so if I use esm to (essentially) transpile on-the-fly, I don't have to distribute CJS versions of my modules. My "index.js" entry point could look like this:

module.exports = require('esm')(module)('./default.js');

That sounds pretty good, although I'm curious about what happens if one of my dependencies is an ESM module.

[jdalton]

My take is: Once you require a build step for interop (publishing 2 versions of your code) you might as well just ship the 1 (transpiled to CJS). It's kind of similar to what we see in browsers with ESM which is "Yes, ESM is neat, but you still need to bundle". Which means that folks are really just shipping transpiled-web-bundles.

The esm loader enables shipping just ESM code. Users don't see esm as a build step and will even refer to it as "enabling ESM without transpiling" (because traditional transpilers are heavier, configier, buildier stepier).

So for your index.js the esm loader can handle 💯 of that load (even deps of deps that have ESM). This would allow you to ship just ESM code and fork for index.js (compat) and default.js (native support). That's super exciting!

[zenparsing]

I tried this approach (using default.js for the entry point and esm for the compatibility layer) yesterday on a little side project and it seems to work very well.

With this approach, we are basically taking an interoperable module graph (as defined above) and downshifting it at runtime to use require. If there are any non-interoperable modules (ones that rely upon asynchronous loading or execution) in that graph, then the downshift operation will fail.

@jdalton I presume this is why esm only supports top-level await for modules with no exports? Because any top-level await in a dependency graph will break the downshift operation?

[jdalton]

@jdalton I presume this is why esm only supports top-level await for modules with no exports? Because any top-level await in a dependency graph will break the downshift operation?

Mainly because it was the easiest way to implement, catch the 99% use case in Node (CLI entry scripts). But yes if I support variant B without the C clause then I'd have to transform it to async-all-the-way-down which means the CJS-bridge would return a promise (not great).

[zenparsing]

But yes if I support variant B without the C clause then I'd have to transform it to async-all-the-way-down which means the CJS-bridge would return a promise (not great).

Do you know if this concern (that generalized top-level-await "breaks" CJS-to-ESM interop) has been brought up to the TLA champions?

[jdalton]

I casually mentioned the promises-all-the-way down bit in our Module WG meeting a month ago or so when we discussed top-level await making it to stage 2 at the May TC39 meeting. Jordan is now aware of the issue too (as it relates to transpilers) and will raise the concern at the next TC39 meeting.