proposal-built-in-modules
proposal-built-in-modules copied to clipboard
tc39
Module specifier: npm-style "@prefix/module" or something else?
Myself and several others at npm (a voting member) feel that we should be using the existing cowpath for scoped module syntax (@std/foo) instead of std:foo or other such alternatives.
While this would usually come down to a simple matter of taste, in this case, we have 254,091 scoped packages in the registry already using this syntax. That's more than most other package registries for other languages, as-is, and is a significant % of our ~950k packages on the main npm registry.
This syntax would also be beneficial because it allows an existing mechanism for polyfills that will work with all current and past versions of node, and can easily be made to work with the browser module system through this proposal and others specifying what non-./ specifiers are supposed to do to resolve.
hey @zkat thanks for chiming in!
For myself I've been hedging on the importance of making an explicit difference between built in modules and ones from the file system. Could you expand on some of the value you see in keeping them the same?
This is currently being discussed in the Node.js repo in https://github.com/nodejs/node/pull/21551, very much appreciate hashing this out and reaching consensus on this and shipping something that is aligned with various bodies / platforms interests.
Another concerns that I had was flexibility in namespaces. Picking the protocol-ish solution would allow us to pick a variety of namespaces without competing with userland, and would be able to be shimmed / polyfilled with import-maps and loaders. If we were to use the scoped module syntax we would then be competing with existing namespaces (the exact problem we are trying to solve in nodejs). Can you imagine a longer term solution that could guarantee flexibility here?
i actually see this as a reason not to use that namespace style. I would want something that sits above npm's (or anyone else's) namespaces, so that namespace ownership is unable to become a problem.
Name collision is a big CON; anyone got the @std scope?
Yes, @jdalton does, but it only contains one package and that package has been deprecated in favor of its non-namespaced version.
(Personally I kinda like the empty string, i.e. @/foo, as a namespace. No risk of collision there!)
Namespace ownership is always going to be a big problem - by using npm's system, we can use their existing solutions for that, instead of immediately finding ourselves in that difficult swamp.
@Mouvedia @bakkot FWIW I've already been pinged by @littledan regarding the std scope and totally willing to donate it to help out the effort if needed.
If the format is @scope/module, you will have fishing going on.
Are we talking about only one std scope or will this be the foundation of many more to come? If it's the former I don't care about the name conflict.
@zkat Thanks for bringing the discussion here. Those numbers are some interesting context.
If we use scheme:module, we might want to register each prefix with IANA as a scheme, to make sure URLs don't reuse the same thing. That process is a bit more heavy-weight than registering a scope in npm, but might be fine if we only have a few. I've heard the suggestion that scheme::module could avoid being a valid URL; maybe that's a way out (and then we can run our own registry!).
Regardless of whether we go with @scope/module or scheme:module, I think import-maps could be used for polyfills (I don't know enough about loaders), couldn't they?
Using a scope looks like a JS module, and using a scheme looks like a special built-in thing. Do we want built-in modules to look special or ordinary?
@littledan using a scope means users can use what they know. Using a scheme means users now have code like:
import {Connection} from "std:worker"
import {map} from "@lodash/functional"
import minimist from "minimist"
And then you need to explain what all 3 do.
users don't know any namespace for standardized functionality from the language because no such thing exists yet. i think many people's first assumption to an import specifier they don't recognize is that it came from npm (for ex. in node when we have people asking about how to install crypto or whatever), which could be confusing.
There are still a lot of additional points that need to be considered here. off the top of my head:
- lots and lots and lots of js users are working in environments where npm isn't used or considered (embedded hardware, game engine addons, etc). what would the best syntax/etc be for someone who isn't using <package manager x>
- given the current monopoly npm has on the js ecosystem's packages (not that this is a bad thing), using a specific namespace that fits for npm opens some doors to potential namespace ownership issues down the road (again i'm not suggesting npm is going to do bad things, but a sun/oracle type situation with the defacto place to put standard polyfills would be bad, and take the ecosystem a long time to recover from)
- various security things (like phishing misspellings of
@std) - how fallbacks are mapped in an environment generic way
- the possible area of conflict for npm with this change (to be 100% clear, i am not suggesting npm has any ulterior motive here, but due diligence is important)
i really like @littledan's idea for a shared repo (possibly via the js foundation?), and a namespace above resolution specific to any platform/tooling would be a fantastic step toward that.
possibly via the js foundation
@littledan that's why I wanted to know what you meant by "we". Even if it's off topic, it's important.
Thanks for bringing this up. Like @MylesBorins, I waffle on whether using scoped syntax for built-in modules is a good or bad idea. Last we talked, he mildly convinced me that different syntax was better.
Let me ask a few pointed questions in the hopes of drawing out folks' intuitions.
Namespace ownership is always going to be a big problem
This doesn't seem obvious to me. For example, if we pick :, then hosts (Node, web, etc.) can completely control the resolution of these spaces, with no overlap with the npm registry. There's not a big problem; instead there's no problem, because collisions are impossible in a centrally-controlled system for governing the behavior of :.
by using npm's system, we can use their existing solutions for that, instead of immediately finding ourselves in that difficult swamp.
What existing solutions are those? The one I'm aware of is emailing [email protected], which makes a judgment call whether to reallocate the scope or not.
The way I see it, the solutions are pretty unsatisfactory and non-applicable for cases that are not actually using the npm registry or filesystem. For example, if we pick @, and later a new IoT product named e.g. blinker wants to introduce blinker@/... modules for its own environment-specific functionality, that introduces several problems:
- Blinker's built-in modules are provided with npm-like syntax, and their users might then expect they could do
npm install @blinker/util, but that won't work, because actually all the@blinker/modules are baked into the Blinker device firmware. - If someone gets wind of the launch of blinker before that company settles on a final name or otherwise reserves the npm namespace, they can cause a confusing situation for people who have such an expectation, by publishing their own
@blinker/utilthat does arbitrary work. (Assume the work is non-abusive and legitimate, i.e. not worthy of a takedown---just confusing for users expecting Blinker's built-in functionality.) - Even if Blinker, Inc. does everything right, they end up reserving the
@blinkerscope on npm only to not fill it with anything. (Because all their built-in modules are baked into the firmware, not distributed on npm.) So we end up encouraging an ecosystem of "scope-squatting" which doesn't even get useful packages published on npm!
And then you need to explain what all 3 do.
Is that a bad thing?
In the particular case of Node.js, should the syntax for accessing files that you control on your local filesystem (i.e. @lodash/functional) be the same, or different, from the syntax for accessing libraries implemented as part of the runtime? Especially as libraries implemented as part of the runtime may change as you upgrade Node, even in backward-incompatible ways.
I'm genuinely curious on folks' thoughts here.
Other points:
If we use scheme:module, we might want to register each prefix with IANA as a scheme, to make sure URLs don't reuse the same thing.
Since module specifiers aren't URLs in most environments, I'm not sure how important this is. (Some URLs are module specifiers, but that's the extent of the relationship.) Similar to the npm scopes issue, assuming that a registry designed for one thing should also be used for built-in module prefixes is bound to lead to confusion and weirdness.
(Personally I kinda like the empty string, i.e. @/foo, as a namespace. No risk of collision there!)
Clever!
Regardless of whether we go with @scope/module or scheme:module, I think import-maps could be used for polyfills (I don't know enough about loaders), couldn't they?
Yes.
I think that Node.js should use @nodejs/fs for its builtins, not nodejs:fs. Also, I think the standard language features belong under a namespace that is reserved by the language, but looks similar; ie, @std/Date rather than std:Date.
People will polyfill and such, no matter what. One value of having node builtins in the same global-space type of pattern (since they predate npm namespaces) is that they allowed userland and platform modules to feel similar to people using them.
One of the values of a lot of the new developments in the JS language (such as Proxies, weakmaps, and so on) is that they allow userland code to do things that were previously only available to the host system. Doing this makes the language feel more integrated and less fractured. I can create my own classes that are iterable just like Arrays, and then-able objects can be awaited. This integration is a very good thing, and it makes JavaScript fun.
This is ultimately a bikeshed, and a purely cosmetic issue about how to express a structured namespace in a module identifier. None is inherently better than the other. But one of them is already adopted by a surpassing majority of JavaScript developers.
The concern about namespace ownership is easily addressed by making @std/ a reserved namespace at the language level. @jdalton is fine with giving it up, and we at npm would be happy to keep people from using it for userland modules (since it'd only cause confusion anyway). Same thing could be done with the @nodejs/ module namespace, just have the host lock it down, and the community will adapt accordingly, since it's a much smaller shift than having to use a whole new naming scheme.
"But what about if some @foobar/ platform comes along, and wants to use that, and there are already @foobar/ modules!" This has already happened, numerous times. There is a module on npm called fs, another called domain. Those were co-opted by the platform, and it wasn't that big a deal, really. Some other node-internal-named packages are used for shimming node-like APIs into browser environments. That's actually a benefit of using the same namespace, not a cost.
This isn't even an example of paving a cowpath. It's already paved. The cows are gone. It's a highway through a city that's 30% bigger than NYC. There's subways and onramps and stuff.
I was midway writing a response but @isaacs' has summarized my feelings very well.
The reason I'm an employee at npm, Inc is that I care a lot about JavaScript and its community. I see JavaScript, the corpus of open-source on npm, Node.js, and modern browser technology, as all part of the same whole.
When we make design decisions that don't look at this collection of technologies holistically, not only do I think it leads to worse products, I think it can serve to fragment the community and create contention.
- module authors should look at trends in language features and use these to shift practices (see: the amazing growth in the language babel has facilitated).
- at the same time, TC39 (Node.js, and other folks helping define platform standards) should look at habits that have grown up in the open-source community and use these to shape decisions
The open-source JavaScript community already has a syntax for scoping packages (@foo/bar) and already has a culture of shimming modules (see @jdalton's esm).
Folks are going to want to shim standard modules, rather than coming up with a way to make this difficult, let's make it possible in an elegant secure way for folks.
The open-source JavaScript community already has a syntax for scoping packages
i think that depends on if you view npm as the entirety of the open source corpus for js code. npm alone has syntax to refer to packages from github (name/repo or git://), gists (gist:id), direct urls or paths to tarballs, etc. and it's not even the only package manager. this seemingly confuses the idea of a "one true namespace"
I think it's important to separate shimmability from built-in module specifiers. At least on the web with import maps, and likely on Node with loaders, you can intercept any arbitrary specifier (including std:foo) and replace it with a shim or polyfill. : vs. @ has no impact on that.
What I'm hearing from the last two posts, if I take out the misunderstandings about shimmability, is that it's more about wanting uniformity between files-on-disks and libraries-shipped-with-the-platform, assuming default usage with no shims or polyfills or loaders or import maps involved.
Which comes back to the questions I tried to ask. Is that uniformity good, or bad?
Which comes back to the questions I tried to ask. Is that uniformity good, or bad?
It is good.
I believe this was implicit in my comment above, which contains much more justification, but tl;dr, yes, that uniformity is good, and without exception, deviations from it are bad.
npm alone has syntax to refer to packages from github (name/repo or git://), gists (gist:id), direct urls or paths to tarballs, etc. and it's not even the only package manager. this seemingly confuses the idea of a "one true namespace"
Arguments to require() and import are not things like git:// urls, though. They're just the names, either of the form foo or @foo/bar, or @foo/bar/path/to/file.js, always.
@isaacs did you mean to post that on nodejs/node#21551?
No, but the same comment could apply equally there. Thank you for the nudge, I will cross-link it.
Which comes back to the questions I tried to ask. Is that uniformity good, or bad?
Stray thought: it would be valuable to compare the approaches taken by other languages, I think. I don't have time right now to do so, but will try to get back to it if no one else gets around to it.
@devsnek certainly there's open-source JavaScript outside of the npm registry. but, npm represents the largest corpus of open-source JavaScript code, and has practices developed across millions of developers.
@domenic to better explain where I'm coming from with the overloaded term shims:
Myself, @boneskull, @iansu, @guybedford, and a few other folks have been exploring how the Node Tooling Working Group can back-port new core-module features, e.g., mkdir --recursive, as they're added to Node.js.
One thought being we'd release modules like @node/mkdir which provide the behavior conditionally on Node version (shimming otherwise). I could imagine a world where we do something very similar for internal modules, e.g., @std/fs is published to npm by Node.js, and would provide a standardized fs module experience across older Node.js versions.
I also like that modules on npm could similarly be published that provide the same behavior as @std/foo in the browser.
☝️ I'd put the ability to do this in the good column.
is this an accurate view of the disagreement?
"npm is super popular/ingrained in the ecosystem, it would be great for users of js to design std modules around it"
vs
"js is used in a lot of places, it would be great for users of js to keep the std modules distinct/disconnected from any specific environment or tooling"
I would like to mention that it's worth to try to separate the standard library from npm. Yes, npm is practically the standard registry, but it doesn't need to be. Also, standard library "packages" are not really packages/modules in the same way a regular module is. As such I would like to recommend not using string identifiers for standard modules.
import { Instant } from temporal;
// insead of
import { Instant } from '@std/temporal';
// or
import { Instant } from 'std:temporal';
I may be wrong, but it seems standard library packages are not supposed to be url resolvable packages. As such, it would make sense that the names be consired syntax instead of urls.
It may be a weird thing for me to say, since it certainly serves my personal and professional interests to equate "npm" with "JavaScript" in whatever way possible, but I'd really like to stop equating this pattern with "npm" per se.
The interesting thing is that millions of JavaScript developers are today using the @foo/bar pattern for namespacing modules. The fact that they are doing this with npm, and that npm (the tool, service, and company) has been a part of establishing this pattern, is somewhat orthogonal to the fact that the pattern is established at this point.
You're seeing people from npm, Inc. getting very passionate about this because we care a lot about modular JavaScript. That's why we choose to work on the problems that we do. But frankly, it ultimately benefits "npm" not at all to do it one way or another, and the pattern isn't tied to npm as a platform in any particular way. Others can and have implemented the same naming convention for JavaScript, and nothing's stopping them from doing so.
So, yes, I agree that it's worth separating the standard library from npm. I also thing that has no bearing on whether the naming convention chosen mirrors that already in use by the vast majority of JavaScript users today.
@bcoe the ability to publish shims, and then configure Node to use them in place of built-ins, works no matter what name you publish those shims under. If you are shimming std:fs, you can publish that as @bencoe/fs and then configure your loader to map std:fs to @bencode/fs. Whether built-in modules use std: or @std/ is really immaterial.
@isaacs you state that being the same is good, but without any reasoning. The rest of your posts seem to just do the same thing, e.g. talk about how it benefits everyone to choose the same naming convention for filesystem-located modules vs. built-in modules, without explaining why. Perhaps you could expand?
@domenic I agree technically we could use either. So it comes down to whether we want users to be aware that these APIs are different to those they import from regular userland sources.
My opinion is that these APIs truly are different AND users should value them differently because:
- they come with a stronger compatibility promise than the average package, e.g. spec, conformance tests
- users ought not to need to do any security/backgrounds check on stdlib APIs as you do when taking third-party code and assessing the trustworthiness of the source (assuming you avoid polyfills)
- there's an implication stdlib APIs are cheaper to load than userland packages, e.g. No network fetch and the VM might even share the native DLL in memory across processes.
- stdlib APIs come with an expectation that they will correctly work in all JS host environments
So let's signal this value to users in the naming scheme. It will aid adoption.
you can publish that as @bencoe/fs and then configure your loader to map std:fs
I think some of my FUD is coming from the fact that how loaders and maps will work, from the perspective of a module author, feels a ways away from being fleshed out. Specifically for overrides to be valuable, it would need to be something that a module author can control (making a library that progressively enhances to new Node.js features) vs., something that a consumer of modules configures.
Also, I was picturing someone would be able to write this code today (pre new loader):
const {mkdir} = require('@std/fs')
And have it already work, if the bridge module has been published..
and write the same code in Node 14 (or some future node), and have it use the built in module.
perhaps I'm just surfacing the requirement that loaders and module mapping functionality be back-ported to older Node.js.
My opinion is that these APIs truly are different AND users should value them differently because...
I think some valuable points have been made by various folks, regarding some of the benefits of making standard libraries stand out like a sore thumb.
at this point i'm confused on if we're talking about node.js builtins or the ecmascript standard library. these two topics, while obviously related, definitely have different constraints and goals, and i don't think we should be directly equating them.
at this point i'm confused on if we're talking about node.js builtins or the ecmascript standard library
I think we are attempting to find a solution that could satisfy both use cases... thus we need to seek consensus around all areas where there might be objections.
One area of contention appeared to be polyfills. @domenic has pointed to import-maps as a solution for browsers and loaders for node.
I've made a proof of concept of polyfilling the namespace nodejs: in node today... this approach could work with any arbitrary string and should be able to be ported to any version of node with a little elbow grease
https://github.com/MylesBorins/namespace-polyfill
As we have a solution for both environments I think it is reasonable to assume that "polyfilling" is a non issue at this point.
What is remaining is a disagreement between whether built-in modules should conform or be obviously different
from @isaacs
There is a module on npm called fs, another called domain. Those were co-opted by the platform
This is not entirely true. We have actually attempted to create a moratorium on shadowing ecosystem packages in node until we sort out namespaces. This has been a significant challenge and has blocked us expanding APIs (independent of if we should be making more apis 😉).
While I am a huge proponent of keeping JavaScript approachable, I think there is value in being explicit at times. The difference between a built-in module and an ecosystem module is something that is very important for a new developer to learn, IMHO.