Feedback on extension resolution

[ljharb]

trafficstars

It seems like a good idea to capture somewhere the feedback we receive on extension resolution.

[ljharb]

https://github.com/nodejs/node/issues/27407

[ljharb]

https://github.com/nodejs/node/issues/27408

[AlexanderOMara]

Is there any information available on why file extension resolution was disabled?

[GeoffreyBooth]

@AlexanderOMara See

• https://medium.com/@nodejs/announcing-a-new-experimental-modules-1be8d2d6c2ff#f606 (the Explicit Filenames section)
• https://nodejs.org/api/esm.html#esm_mandatory_file_extensions
• #268

[sheerun]

My feedback is that:

1. When I write import ./foobar then of course I want to import js file, not mjs, cjs, wasm, css, json, or other. This is also "what users are familiar with" as you noticed. If you want to introduce explicit extensions then please introduce them for all other file extensions than js.
2. Nobody is ever going to use url imports like https://unpkg.com/[email protected]/src/Three.js in Node. It's just silly thing in many aspects like reliability and security.
3. You can match browser behavior but it shoudn't prevent you from making convenient defaults for Node developers. After all you do it import so it supports resolving bare paths from node_modules. Why wouldn't you make good defaults for file extension as well? (i.e. resolve to js by default)
4. As node module author if I want to use modules I shouldn't be forced to change file extensions to mjs. Specifying in my package.json that js files in my module use ES6 modules (or specifying entrypoint to ES6 modules file) should be enough. Most node module authors already do it this way and node is late to the party to suggest some other way, especially totally not backward compatible with status quo and basically requiring module authors to rename every file in their repositories.
5. "There’s a performance cost to automatic file extension/directory index resolution": import doesn't do that anymore as described in this blogpost. It just looks for single field (package.main). Having js extension a default when no extension is provided also doesn't make performance worse.

So for the record I think forcing on community yet another .mjs and .cjs extensions was mistake to begin with and complicates already complicated landscape. Per-package configuration with package.type is already how it works so why on earth you require to confirm this additionally through renaming all files in repository. It's not like module authors will have half .mjs files and half .cjs files because even offical node docs now say we can't and shoudn't "mix them". Either all files in project will be in ES6 format or CJS format so no need for "file extensions" granurality.

[AlexanderOMara]

UPDATE: I think this is actually the biggest problem: https://github.com/nodejs/modules/issues/323#issuecomment-511268264 , https://github.com/nodejs/modules/issues/352

The below, while not super pleasant, could be worked around with smarter transpilers relatively easily.

---

Before, it was possible to write ES modules (or TypeScript modules), and publish both CommonJS and ES modules with a simple module transformation via babel (I've actually started doing this already, so my modules can be treeshaken).

Now we can't write:

import {foo} from './bar';
console.log(foo);

We have to write:

import {foo} from './bar.mjs';
console.log(foo);

Which means that now any transpiler also needs to rewrite the actual import path. I don't think any currently have this functionality, because until now this wasn't an issue.

This is very different from how the ecosystem has worked so far. For example, TypeScript and Webpack both work just fine with the extension-less module resolution. I would guess rollup does too.

---

I thought the previous design made much more sense.

When I import something, naturally I expect to get an ES module, preferring the .mjs extension, probably falling back on a .js CommonJS module exported as an ES module.

If I require something, naturally I expect to get a CommonJS module, preferring the .js extension which can't really be changed for backwards compatibility, probably falling back on loading a .mjs ES module exported as a CommonJS module.

I get that some people don't like the new file extension, but I don't think it makes sense to create new problems and add ambiguity just to keep it.

[MylesBorins]

Some feedback in favor of not having extension searching on by default

Twitter poll about compat: 65% of 1295 people favor Browser compat

@jhnns on twitter: https://twitter.com/Jhnnns/status/1003201464716726272

@mhart on twitter: https://twitter.com/hichaelmart/status/1039529625100185600 @brianleroux on twitter: https://twitter.com/brianleroux/status/1039653429272952832

Announcement tweet for PR had 150k impressions and no negative feedback

Tweet asking for feedback

Obviously none of this is scientific... but thought I could offer some balance to the feedback

[ljharb]

Again tho, enabling extension resolution does not in any way prevent “browser compat” - this is purely a question of whether you want the preferences of one group (browser compat folks) to oppress another (back compat folks) by making the feature off by default - because on by default causes no damage, but off by default does.

[AlexanderOMara]

Is browser compatibility really even possible? As soon as you import another module, isn't browser compatibility lost?

import {something} from 'some-module';
console.log(something);

In node, that would have to resolve to something in node_modules right? You're not expected to write import {something} from 'node_modules/some-module/index.mjs'; right?

Am I missing something here?

[hybrist]

@AlexanderOMara Some browsers have shipped support for import maps which do make the code above work, assuming the appropriate import map is provided that tells the browser where some-module lives.

[MylesBorins]

@ljharb I disagree with your claim that turning it on by default has no undesirable outcomes. I also think we can avoid boxing people into categories and creating an us vs them narrative. We all care about our ecosystem and making a great developer experience.

@AlexanderOMara the import map proposal

Chrome platform status: https://www.chromestatus.com/feature/5315286962012160 Tracking issue: https://bugs.chromium.org/p/chromium/issues/detail?id=848607

[AlexanderOMara]

I think it would help a lot if the proposed changes to the ecosystem were better fleshed out and documented.

• From a package author's perspective, what are the implications?
• From a transpiler's or to-JS compiler's perspective, what new features are needed?
• From a user's perspective, what are the advantages and disadvantages of this change?

I don't think the changes are just about browser-support vs backwards-compatibility. As I understand it, there would also be a loss of certain functionality, but potentially also a gain of new functionality.

At this point, I'm honestly not sure which system I would prefer once all the pieces are in place, and currently I think it's a lot easier to see the negative implications of this change.

[GeoffreyBooth]

Let’s be clear: we shipped automatic extension resolution in ESM. It’s --es-module-specifier-resolution=node. It’s just not on by default. The people asking for extension searching could simply be told to use that flag to turn it on. It’s an inconvenience, sure, but fairly minor in my opinion. The question is whether that inconvenience is justified by what’s gained from making searching disabled by default.

As far as I can tell, to @AlexanderOMara’s point, the biggest consequence of having searching disabled by default is that public package authors can’t assume its availability—which is a big benefit, in the eyes of many in this group, as that encourages packages published to public registries to be cross-compatible with browsers by default (at least as far as resolution is concerned). That’s what would be lost if the default is flipped. To me, that benefit outweighs the cost of users needing to use a flag if they want this enabled in their projects.

From a package author’s perspective, what are the implications?

They need to publish their packages with import specifiers that include file extensions. This will annoy some authors. However, such packages will work in browsers without extra transpilation or building, assuming they aren’t otherwise incompatible (like by importing native modules or Node APIs that can’t run in the browser). Even if a particular package author is exclusively targeting Node, all the package authors who are targeting browsers (and/or Node) benefit from a broader ecosystem of more cross-compatible packages, as that expands the number of potential dependencies that a browser-targeting package can import.

From a transpiler’s or to-JS compiler’s perspective, what new features are needed?

None. Transpilers can add the ability to add extensions at compile time, so a specifier like './file' can be rewritten to './file.js' during compilation along with whatever else is getting converted by the transpiler. This would provide similar UX without needing the flag, if the user didn’t want to either just type the extension or use the flag.

From a user’s perspective, what are the advantages and disadvantages of this change?

See above. They need to use a flag if they like this behavior, so that’s potentially a disadvantage. The advantage is that if they use any publicly published packages in a browser context, those packages are easier to work with in that they don’t require a build process or a specially configured server.

[ljharb]

They're already not cross-compatible with browsers because they use bare imports. import maps work the same way with bare imports as with extensions.

In other words, unless we ban bare specifiers altogether, "work in browsers without extra transpilation or building" either is a) identically true with or without default extension resolution, or b) is identically false with or without it.

[AlexanderOMara]

From a user’s perspective, what are the advantages and disadvantages of this change?

See above. They need to use a flag if they like this behavior, so that’s potentially a disadvantage. The advantage is that if they use any publicly published packages in a browser context, those packages are easier to work with in that they don’t require a build process or a specially configured server.

Suppose I'm making a module that's explicitly node-only. What do I have to do to load a native module? Do users have to add a flag just to use it?

Suppose I'm making a module that has a WASM component. Does that mean using node-only functionality, or is there a way to do that in a browser-compatible way?

[bmeck]

Suppose I'm making a module that's explicitly node-only. What do I have to do to load a native module? Do users have to add a flag just to use it?

In general the ability to interpret different formats are not affected by resolution, so thats out of scope of this particular issue. However, the list of supported module formats could differ between Node and the browser and both have ways to intercept requests and translate to a different supported format (though I doubt the cost of doing so in the browser is worth it).

Suppose I'm making a module that has a WASM component. Does that mean using node-only functionality, or is there a way to do that in a browser-compatible way?

WASM requires all APIs be passed in via imports, I don't understand the question. Both browsers and Node are looking at supporting loading WASM format resources. This also seems out of scope of this issue.

[GeoffreyBooth]

Re import maps, see https://github.com/WICG/import-maps#extension-less-imports:

It is also common in the Node.js ecosystem to import files without including the extension. We do not have the luxury of trying multiple file extensions until we find a good match. However, we can emulate something similar by using an import map. For example,

 {
   "imports": {
     "lodash": "/node_modules/lodash-es/lodash.js",
     "lodash/": "/node_modules/lodash-es/",
     "lodash/fp": "/node_modules/lodash-es/fp.js",
   }
 }

would allow not only import fp from "lodash/fp.js", but also allow import fp from "loadsh/fp".

Although this example shows how it is possible to allow extension-less imports with import maps, it’s not necessarily desirable. Doing so bloats the import map, and makes the package’s interface less simple—both for humans and for tooling.

This bloat is especially problematic if you need to allow extension-less imports within a package. In that case you will need an import map entry for every file in the package, not just the top-level entry points. For example, to allow import "./fp" from within the /node_modules/lodash-es/lodash.js file, you would need an import entry mapping /node_modules/lodash-es/fp to /node_modules/lodash-es/fp.js. Now imagine repeating this for every file referenced without an extension.

As such, we recommend caution when employing patterns like this in your import maps, or writing modules. It will be simpler for the ecosystem if we don’t rely on import maps to patch up file-extension related mismatches.

[AlexanderOMara]

Suppose I'm making a module that's explicitly node-only. What do I have to do to load a native module? Do users have to add a flag just to use it?

In general the ability to interpret different formats are not affected by resolution, so thats out of scope of this particular issue.

I had thought the ability to import things that aren't JS modules was part of this issue. Maybe I was mistaken.

[ljharb]

@GeoffreyBooth sure. and i think it's a mistake that the import maps proposal isn't yet providing a more flexible means to handle that use case - but that's not something that should make node constrain itself.

[bmeck]

@ljharb I do not see this as a constraint, but rather a useful step in iterating so that we can ensure the end result of our process is what we desire step by step. I think going conservative with our ecosystem forwards compatibility is better than diverging for backwards compatibility. This is an issue about feedback but saying we are being constrained makes it sound like feedback about why extension resolution is useful wouldn't be accepted. We can iterate and improve our modules implementation over time :).

[MylesBorins]

@AlexanderOMara importing things that are not js is not the issue. We have experimental support for JSON and a PR is open for WASM. We are also doing work with browser vendors and the wasm spec authors to standardize the same modules for browsers 🎉.

The bigger issue is that browsers will never do multiple network calls to resolve a file extension, so having specifiers throughout source text without file extensions creates a universe of "node only" code. It could be transpiled, but why should it have to be? Yes, people can always choose to write code with a subset if they wish to, but my feelings on the matter is that we will over time have a more stable cross platform ecosystem if we minimize the places where things diverge.

If you choose to use Node.js apis, you will find yourself in a place where your diverge... but it is possible to polyfill those specific APIS (see browserify).

[ljharb]

@bmeck because of extensionless flies and type module, I’m not convinced that it won’t be a breaking change to add default extension resolution later.

The module system is not like other things; it can’t really be iterated on over time. Just like cjs, whatever we initially ship may be effectively the entirety of the system for the foreseeable future.

[bmeck]

@ljharb

@bmeck because of extensionless flies and type module, I’m not convinced that it won’t be a breaking change to add default extension resolution later.

If this is considered breaking, it just means people could opt-in via a flag, like package.json or w/e.

However, we can be more concrete here with an example. The addition of an extension-less file in a place that it collides requires an invalid specifier:

/foo
/foo.js

import '/foo' would still find /foo before /foo.js with current algorithm for extension searching. Meaning the confusing case would actually be import '/foo' with:

/foo.js

Which would error without resolution. Moving from error to non-error seems ok to me and often is not considered breaking when adding features (particularly when some feature is missing like an API).

The module system is not like other things; it can’t really be iterated on over time. Just like cjs, whatever we initially ship may be effectively the entirety of the system for the foreseeable future.

I disagree, CJS has evolved (slowly) over time. ESM can do the same. Whatever initially ships might want to setup forward compatibility paths if desirable, but has no clear reason it is unable to match CJS' evolution over time. Part of the problem with CJS is the outstanding number of features and dynamic behavior that it exposes publicly, and we are not looking to duplicate that to my knowledge in the ESM implementation. If there are clear paths we want to reserve, we can do so; however, without clear reasons conservative iteration seems a good path forward.

[ljharb]

Why would an extensionless file be selected over an extensioned one, if we have extension resolution?

[bmeck]

@ljharb because thats how existing extension resolution works (see LOAD_AS_FILE); this had the historical reasoning that it should check the exact specifier first since it could be a real file on disk and the code did explicitly request that specific string so it should be checked before doing any magic.

[rauschma]

Subjective feedback from someone who has used the latest ESM implementation in a few experiments recently: I really like how it works. I appreciate its simplicity and find it easy to understand and to explain (easier than CJS). I think having fewer automatisms (“magic”) is a benefit.

[SMotaal]

@ljharb please elaborate on what you meant here:

import maps work the same way with bare imports as with extensions

Specifically, how you see importmaps not working for "extensionless mappings" to "locations having the actual extension" — or whatever I might have misread.

Note: I'm thinking here of the symmetry of export maps for the same when they move ahead.

[ljharb]

@SMotaal you can map from lodash to path/to/lodash.js the same way you can map from lodash/foo to path/to/lodash-foo.js. The only tradeoff is that potentially the size of the import map could be much larger if two things happen: 1) default extension resolution and 2) no corresponding import map feature ships to adjust to this legitimate use case.

[SMotaal]

I see, so I think it makes sense for authors of importmaps to adhere to one style over the other to avoid extra moving parts and complexity.

[ljharb]

I’m not sure what that means - tools, not people, will likely be authoring most import maps, and the style can vary as long as there’s validation tooling (which there will be)