proposal-built-in-modules

proposal-built-in-modules copied to clipboard

Should built-in modules be all with one scope/scheme prefix in their module specifier, or should we have multiple prefixes based on what kind of module they are? I've heard different opinions from different people, so I though it could be good to collect the arguments all in one thread.

Shared namespace

We could put all built-in modules with a specifier like "@std/module" or "std:module", regardless of where they come from or what they're for.

Advantages:

• It's often not clear how to categorize a module, as modules may be gradually introduced into different embedding environments over time, with an attempt to match each other.
• Coordination in use of this namespace can take place either implicitly--if nothing else, browsers will eventually implement things and notice overlaps--or explicitly, e.g., through something like https://github.com/littledan/js-shared-interfaces/blob/master/MODULES.md .
• Developers don't really think about standards bodies when programming, and standards bodies don't really cleanly map onto embedding environments or logical spaces in practice, so it would be confusing to subject them to all that.

Split namespace

Several different namespaces for built-in modules could be used, such as "@js/", "@html/", "@nodejs/" (or, "js:", "html:", "nodejs:"), for different modules.

Advantages:

• Different standards bodies or embedding environments could use different prefixes, reducing the need for coordination between them, so they can each create new modules independently.
• The prefix could be used indicate which subset of environments a module is valid in, making it more apparent how to write code that works across multiple environments.
• When multiple environments have minor differences between the semantics of a similar module, they can use the same name with a different prefix, to indicate the difference.

Next steps

What are your thoughts?

My current feeling is that the web + JS should use a single namespace, but other environments like Node.js might use additional namespaces for Node-specific functionality.

I think that the Web should just be an environment where JS is used, and it should not be treated special in regard to namespaces. I think everything that is web-specific (web standards) should be namespaced.

I'm trying to wrap my mind around this.

I think it makes sense for Node.js but I'm not sure how it would work for other environments. Here's my best guess:

import { Promise, Worker, Atomics, ArrayBuffer } from '@std/common';
import { document, window, DocumentFragment } from '@std/dom';
import { writeFile, readFile } from '@nodejs/fs';
import cluster from '@nodejs/cluster';
import { app, BrowserWindow } from '@electron/common';
import desktopCapturer from '@electron/desktopCapturer';

Is Electron an environment or should it stick to user-land modules?

I don't think there's any particularly reasonable slicing of things for the web platform to use separate namespaces, so it should all be on one namespace. (Plus, in general, separate namespaces means I'll have to remember two names for each API - the actual name, and the namespace that is arbitrarily associated with it.

and the namespace that is arbitrarily associated with it

That's not a legitimate worry unless you have an example of a module that would be warranted in two different scopes. If you know what you are importing the scope is obvious.

URL is warranted in both browsers and node, but not in a context without an internet connection (like an embedded JS engine in a device).

I think that's off topic but in this case Id just mirror it on the other scope.

@ljharb In the case of URL how do you suggest this scenario is handled? Does the embedded JS engine avoid exporting URL or does URL need to be in a different namespace such as @std/net because it's not "common" enough?

import { Promise, Worker, Atomics, ArrayBuffer } from '@std/common';
import { document, window, DocumentFragment } from '@std/dom';
import { URL, fetch } from '@std/net';

@styfle I think that's why having things be split isn't a good idea - if there's only a single namespace, then the user doesn't have to guess about what category the thing they want is in - they just import it, and if it's not there, it'll error out prior to evaluation.

I think having multiple namespaces is fundamental for the ecosystem is governance and standardization. Who owns/organizes the non-prefixed namespace? Who owns/organizes the std namespace? I think this is far more important than the actual syntax.

If it becomes a recommended pattern that a runtime/platform can implement/add its own runtime specific part of the std lib, then it negates the benefits of having a std library in the first place. If the goal for the standard library is to include browser specific APIs, then it would be a huge loss for the ecosystem for non-browser JS implementations. I think the only solution is to have multiple namespaces, so it’s clear for the user what that API is about. It also makes polyfills easier, because it’d be clear where there are tradeoffs.

Let me make an example: having URL in the standard library would be a great win as it’s a simple data structure. Having fetch would become problematic for non-browser environments, mainly because fetch has some defined semantics regarding caching, security and connection management that make sense only if the JS vm is associated with a single user.

I don’t think having multiple namespaces would be a huge problem for the community: people need to read the docs anyway and most editors offers great autocomplete.

I lean towards a shared namespace but to make an educated guess Id need at least the first 10 or so modules that will be available once std is implemented. If you are planning on organic additions, that might clash with the chosen approach if the eventual modules are not the ones expected.

In that regard, Id recommend to go with the unified approach as the desired outcome and if it fails, allow/enable the split if it's really warranted.

TL;DR: your first goal should be to make std worth using and design it as a template to enable the possibility of other environment-specific scopes.

I think the issue of how to manage the shared namespace could become more significant if we increase the velocity of what we standardize, which I hope this effort will enable. However, I agree with many points made here about the benefits of a shared namespace.

We've also been using names which are less likely to overlap--in web standards, it's often a single additional global whose name starts with something like Web, and in TC39, we often avoid creating additional globals. Some new web standards (especially from WHATWG) play more in the space of ergonomic names, which is great, and I hope standard modules give the freedom to continue in this direction.

So far, the governance of the shared global object namespace has been implicitly based on browsers: any overlaps get noticed in their development process by the time they're looking into implementing and shipping something, if not before. So far, this has worked just fine, but it's worked in an environment with less potential contention by design than we might have in the future.

Do we want to continue this browser-based namespace governance, or should we do something more explicit to coordinate between standards bodies to identify potential conflicts earlier in the design process? I'm optimistic that we can cooperate here. To the people who want split namespaces: What challenges do you see in cooperating here?

So far, the governance of the shared global object namespace has been implicitly based on browsers: any overlaps get noticed in their development process by the time they're looking into implementing and shipping something, if not before. So far, this has worked just fine, but it's worked in an environment with less potential contention by design than we might have in the future.

The current global namespace recommends every runtime to add its own things to it, meaning that there is no single source of truth to what you can expect being global in a JS environment. The benefit of a standard library is that it is ubiquitous. If every different JS environment can add to the standard library things that could not be implemented by a generic JS runtime (they are specific to a browser or a server for example), i.e. if pieces of the standard library become optional, then the benefit of having a standard library collapses for an ergonomic point of view when creating isomorphic code.

Multiple namespaces enable the web to standardize all the APIs that make sense in a Browser context and at the same time it enables non-Browser environments to not do so. As an example, the DOM could be standard, and there will no question for an IoT runtime to have the DOM or not. Having multiple namespaces (or sub-namespaces) acknowledges the fact that JS can run in non-browser environments. Developers will know from looking at the import if a module is expected to be there or not. In turn, this create less pressure for non-browser runtimes to not behave like a browser as those modules will be "web modules".

I think the standard namespace should be limited to utilities of the core part of the language and do not assume where something could be run. Would you expect the engines to include full HTTP/HTTP2/HTTP3 capability, a database, a caching layer, a file system API? I think the greatest thing about JS that enabled so much creativity and innovation was that none of those things where specified.

I don't see what that has to do with shared/split namespaces. What if we list the pieces of the standard library that are required in the JS spec, and make sure to document what interfaces are supported in what environments carefully, e.g., in MDN?

What if we list the pieces of the standard library that are required in the JS spec, and make sure to document what interfaces are supported in what environments carefully, e.g., in MDN?

That could potentially work, but it would create a lot of confusion on why a certain module is not in a given runtime as it will be buried at the end of a MDN page. There are JS standards and Web standards, this difference is important to developers and they should be immediately aware of it. Moreover creating a namespace mechanism enables runtimes to add custom or experimental behavior in a safe way without any potential collisions.

I think adding a standard library is the perfect moment to reduce the friction when writing isomorphic applications. If a developer needs to look at the bottom of an MDN page to know where that part of the standard library work, then this mechanism is only slightly better than adding globals.

Looking at https://github.com/denoland/deno_std I think @ry might have an opinion on the matter.

I'm totally in support of reducing the friction when writing isomorphic applications. I don't think sprinkling prefixes throughout JS source implying, "things defined outside of TC39 are not available for isomorphic code" is the best way to do that, since it's already not true.

I think @mcollina is making some good points, and perhaps I can add to it with a concrete example.

Certainly we want to import temporal (a language-level/isomorphic builtin) like so:

import { ZonedInstant } from 'std:temporal';

Good so far.

From a strictly web/browser point of view, it does make sense to share the namespace:

import { asyncLocalStorage } from 'std:async-local-storage';

OK, that seems fine.

But if we extend that methodology to Node:

import { readFile } from 'std:fs';

That just doesn't look right at all! It feels much more appropriate to have something similar to:

import { readFile } from 'node:fs';

Taken together, the situation appears to be that we have priveledged the web and browsers over Node.

Is that a good thing?

Why does it make sense to you to put localStorage in the standard namespace, but not fs? To me that seems like implicit browser bias - I think it either makes sense for neither to do so, or both - iow, both being in the standard namespace "look right" to me.

Taken together, the situation appears to be that we have priveledged the web and browsers over Node.

Is that a good thing?

As both JS and the Web Platform are based on specced standards, and Node not necessarily - I guess that is fine

Thanks for posting this. I had not been paying attention to this thread, so I don't know the particular issue yet (will read). But on the general issue stated here, I think it would be terrible for us to privilege the browser/web over Node, or over any of the increasing number of other host environments (IoT, blockchain, ...)

Just dipping in, but the problem with 'std:async-local-storage' is: which standard? Say Node were standardized. 'std:fs' would still be wrong. You folks are all too acclimated to the web. To those of us focused on other uses of js, 'std:async-local-storage' is just as wrong.

@erights I think the idea with a standard library is low level apis that for sure exist in all environments. Similar to what already exists, but not all bundled in the global context

You folks are all too acclimated to the web.

i don't see anything wrong with prioritizing web use-case, as it reflects industry reality. pretty much all javascript software-development in industry deals with (or is ultimately meant to support) passing workflow-data to/from web.

That’s simply not the case any more, even if it may have been at one time.

pretty much all javascript software-development in industry deals with (or is ultimately meant to support) passing workflow-data to/from web.

That is nowhere even close to true.

A thing that would help here is taking ArrayBuffer, pretending it were invented in a namespaced world, and explaining the effects of that in a split namespace system. What would be the expected outcome for such a feature and how costly is that for all parties involved?

I would prefer splited namespace because I expect proprietary JavaScript embedding to expose some of their functionalities using modules within a non standard prefix.

If we do share the same prefix we could end up with collisions and incompatible APIs.