Latest WebAssembly ESM Integration rebase

[guybedford]

WebAssembly ESM Integration including Source Phase Imports

This represents the latest rebase of the ESM Integration spec, replacing the original PR in https://github.com/whatwg/html/pull/4372. Major changes since then include source phase imports support and instantiation being updated to be synchronous.

• [x] At least two implementers are interested (and none opposed):
  • Chromium (with V8 source phase design doc)
  • Node.js
  • Deno
  • Previously WebKit, pending impl refresh
• [x] Tests are written and can be reviewed and commented upon at:
  • Initial ESM Integration: https://github.com/web-platform-tests/wpt/pull/32862
  • Source phase support: https://github.com/web-platform-tests/wpt/pull/42467, https://github.com/web-platform-tests/wpt/pull/45401
  • Tentative -> Non-tentative PR: https://github.com/web-platform-tests/wpt/pull/46556
• [x] Implementation bugs are filed:
  • Chromium: https://issues.chromium.org/issues/40244733, with source phase support tracked in https://issues.chromium.org/issues/42204365
  • Gecko: https://bugzilla.mozilla.org/show_bug.cgi?id=1899832
  • WebKit: https://bugs.webkit.org/show_bug.cgi?id=274908
  • Deno (only for timers, structured clone, base64 utils, channel messaging, module resolution, web workers, and web storage): https://github.com/denoland/deno/issues/2552
  • Node.js (only for timers, structured clone, base64 utils, channel messaging, and module resolution): https://github.com/nodejs/node/issues/53178
• [x] MDN issue is filed: https://github.com/mdn/mdn/issues/549
• [x] The top of this comment includes a clear commit message to use.

(See WHATWG Working Mode: Changes for more details.)

---

/acknowledgements.html ( diff ) /indices.html ( diff ) /infrastructure.html ( diff ) /references.html ( diff ) /webappapis.html ( diff )

[guybedford]

For the question of why Wasm and JS are treated as equal capabilities in the module system, the reasoning here is that unlike other resource imports which are implemented as synthetic modules, Wasm modules are implemented as full cyclic module records that can in turn import any other JS modules. Therefore while Wasm itself is usually a securely sandboxed language, as soon as you have arbitrary imports to JS, you have equal capability since any Wasm module imported this way can import and therefore execute arbitrary JS in turn.

[guybedford]

Thanks for the review, I've addressed the changes in https://github.com/whatwg/html/pull/10380/commits/f816de8a0288b58a01b26796aea1b878d1b51087.

[guybedford]

I've integrated the latest changes here so we should be good to go.

Just let me know if I can do anything to help get the participation check passing.

[domenic]

Just let me know if I can do anything to help get the participation check passing.

It looks like you signed up as an individual at some point, but you work for Fastly, which has signed up as an entity. However, Fastly has only signed up to contribute to Fetch.

So this is a bit complicated. The ideal steps needed are:

• @fastlyneal needs to send a PR to update this part of entities.json to include the HTML workstream. (He might also want to remove mnot as a GitHub contact there, since mnot appears to no longer work at Fastly...)
• You can send a PR to remove yoursself from https://github.com/whatwg/participant-data/blob/main/individuals.json
• You need to join, or make public your membership in, the Fastly GitHub organization.

[guybedford]

The participation check is now passing. @domenic thanks for the clear help with this.

[domenic]

This looks ready to merge to me, but I'm not 100% sure that we have multiple browsers interested in implementing exactly this shape of the proposal. We have zero browsers passing the tests, for example: https://wpt.fyi/results/wasm/webapi/esm-integration?label=experimental&label=master&aligned

[annevk]

Although the WebKit team at Apple was initially somewhat skeptical about bundling JS and Wasm as a single destination, we're okay with that now under the expectation that the privileges and capabilities of JS and Wasm won't diverge.

Who can speak for Chromium, maybe @syg?

And Gecko would be @codehag or @mgaudet perhaps?

[codehag]

This looks ok from our side.

[guybedford]

@ajklein may also be able to speak to the current progress on the V8 implementation here.

[ajklein]

Speaking to implementation progress: in V8/Chromium, we are currently working on supporting source phase imports for loading WebAssembly modules as source (tracked in https://issues.chromium.org/issues/42204365). We are not currently prioritizing work on full ESM/Wasm integration. Essentially, we're working on item (1) of the Wasm ESM Integration proposal's Progressive Implementation Support section.

[syg]

IIUC, this PR adds Wasm support for both source phase imports (e.g. import source foo from 'bar.wasm') and evaluation phase imports (e.g. import 'bar.wasm'). As @ajklein says in https://github.com/whatwg/html/pull/10380#issuecomment-2269980165, V8/Chromium has planned work on source phase imports only.

It's also my understanding that the WPT tests linked above test both source phase imports and evaluation phase imports.

@annevk Is WebKit planning to implement and ship Wasm evaluation phase imports as well as source phase imports?

@codehag What about Gecko?

If the answer to both questions above is yes, then seems fine. Otherwise I feel that we should separate the source phase support from the evaluation phase support, as we shouldn't merge something that won't (fully) reflect reality for a while.

[codehag]

Thanks for the ping @syg. My understanding is there are two parts to your question:

1. Will Gecko Support both source phase imports and evaluation phase imports for wasm?

If they are both standardized, our plan is to support both. The work is not yet scheduled.

2. Given the changes to the spec in recent years, source phase imports make more sense for wasm. Should we support both source phase imports and evaluation phase imports right now, or should we wait on evaluation phase imports to see if they have a use case?

This PR was initially specifying evaluation phase imports, before source phase imports were discussed in TC39. Evaluation phase imports have been supported by webkit behind a flag for some time IIUC. However, most people importing wasm today are doing so as "source" first, and then instantiating it themselves. This was a major motivation for source phase imports in TC39 to begin with. This begs the question: If evaluation phase imports are specified, will they actually be used? Or are we supporting them here purely out of spec inertia? I don't actually know the answer here myself.

Consistency between JS and WASM importing would be nice, especially since they will be using the same syntax. I could see some users experiencing confusion if imports between WASM and JS are not equivalent. Maybe @guybedford could speak to whether evaluation phase is important for users?

We (Gecko) could go either way. It may make sense to wait just because source phase is more useful. But, if the answer to "will evaluation phase imports be used" is that "yes, evaluation phase imports are important and we won't learn anything new if we wait", then we might as well spec this now and gradually adopt as per the plan in the Progressive module support document.

cc @eqrion @dminor for visibility and any further comments.

[syg]

Thanks, @codehag. The "we will support if standardized" is a bit entangled so let me try to disentangle that AFAIU:

• Evaluation phase Wasm ESM integration is not yet standardized / ready to implement. Whether evaluation phase Wasm ESM integration is standardized is being done in Wasm CG, not WhatWG.
• If Wasm CG standardizes evaluation phase imports, we need HTML support.
• Source phase Wasm ESM integration is already Stage 3 in TC39.
• We need HTML support for source phase imports.

IOW, my understanding of this HTML PR is purely to support proposals standardized elsewhere. We don't really need to get into the actual motivations for the other proposals here, there're other venues for that already. But because this PR is purely about support, I want it to reflect the most likely interoperable future.

Best I can tell, source phase imports is definitely in that future. I don't know if evaluation phase imports is in that future yet. Chrome isn't working on it right now, thus the question to Mozilla and Apple if you have a different opinion.

[nicolo-ribaudo]

Yes, this PR adds HTML support for wasm imports regardless of their phase (source vs evaluation). Which phases are supported is entirely controlled by the Wasm proposal, which defines the WebAsssembly Module Record.

In general, import phases have been designed to be opaque to HTML's loader, and only "visible" to ECMA-262 and whoever provides custom Module Record types (it's this obviously visible to HTML if it for CSS modules, since they are defined here).

[guybedford]

While the motivation for source phase imports remains that it is considered the more useful route for the instantiation of WebAssembly modules, this does not negate the need for or usefulness of the instance / evaluation phase even if those are considered a much smaller use case at this point. There are many packages on npm today that completely work with standards-compatible ESM evaluation phase integration.

There are no plans to deprecate or remove the evaluation phase. The standardization of the ESM Integration spec at the Wasm CG into Phase 3 included both phases, with the original spec mechanics and WPT tests for the evaluation phase remaining throughout the source phase process per the Webkit implementation, with just a couple of small changes (mainly moving to synchronous instantiation). Per the README note linked, the weakening of the implementation constraint to allow shipping one without the other specifically does not indicate that either should not ever be implemented - the standard includes both phases and that is what is expected to be implemented.

Node.js ships the instance phase under the --experimental-wasm-modules flag already. For Node.js, our plan once V8 source phase syntax is shipped in Node.js is to then unflag with both the source and evaluation phase supported.

As @syg points out, the conformance decision here is entirely made in the ESM Integration spec though. We could possibly add some further clarifications on the HTML side to note both phase are specified if that would be useful.

[syg]

Yes, this PR adds HTML support for wasm imports regardless of their phase (source vs evaluation). Which phases are supported is entirely controlled by the Wasm proposal, which defines the WebAsssembly Module Record.

This was part of my question: is it exactly the same? That is, if you were to limit this proposal to just supporting source phase imports, can you remove stuff from this PR? E.g. this PR talks about module requests for WebAssembly Module Scripts, which isn't needed for source phase imports, right?

[nicolo-ribaudo]

That step there also affects import source. i.e. this will result in an error (assuming that there is no import map mapping "not a valid URL" to something):

import source foo from "./x.wasm"

(module
  (import "not a valid URL" "num" i32)
)

That however raises indeed a question: is this intentional? @guybedford

[nicolo-ribaudo]

I talked a bit with Guy about that, and no that is definitely not intentional. Doing import source x from "foo" should not validate "foo"'s imports (regardless of whether foo is a WASM source, of a JS source if/when we'll add them).

The reason HTML has that check is that we want to abort the loading process as early as possible when we know that it will fail: in case of

import "./valid.js";
import "invalid specifier";

we want to avoid starting the network request for ./valid.js, since we can easily tell that the process will fail anyway.

It seems like we can get the same behavior by moving those from "create a XXX module script" to the beginning of HostLoadImportedModule, validating referrer.[[RequestedModules]] before doing anything else. There is no observable work happening between when "create a JavaScript module script" returns and when the first HostLoadImportedModule call for its dependencies happen, so the only difference is that we'd do this pre-validation only when we know that we want to load the dependencies of that module.

[guybedford]

Thanks @nicolo-ribaudo for bringing up this extra validation step - as we discussed, I've added a refactoring in https://github.com/whatwg/html/pull/10380/commits/da7425ed678648d18dc17ad517e5410ac98868d4 to move the validation into HostLoadImportedModule so that source phase imports no longer validate specifier resolution since they are supposed to remain virtual.

That is, if you were to limit this proposal to just supporting source phase imports, can you remove stuff from this PR? E.g. this PR talks about module requests for WebAssembly Module Scripts, which isn't needed for source phase imports, right?

@syg this led to a great catch - with the latest validation refactoring which can be considered a simplification in only performing validation and resolve validation once during the main load algorithm, yes this PR now remains mostly equivalent with and without the source phase integrated on the Wasm spec side. I've also added an additional note in the last commit to mention the source phase explicitly and to avoid any ambiguity in https://github.com/whatwg/html/pull/10380/commits/6a5cc63eca2a27305edbdfeecd3df492227057c8, let me know if that helps?

[nicolo-ribaudo]

It seems like there has been a problem with the PR preview. Could it be re-triggered? It would make reviewing much easier :)

[annevk]

If you edit OP it makes PR Preview run again. I just did that and it succeeded.

[syg]

I'm not too familiar with the HTML side of the module loading machinery but I think this makes mostly sense to me now, and I'm happy with it landing. Recapping my understanding of the state of the world:

1. Wasm ESM integration is specced across three (3) specs: HTML, ecma262, and the ESM integration proposal.
2. This PR adds the ability to fetch and parse Wasm modules. With the HostLoadImportedModule refactoring, it is agnostic to the phase of the import.
3. What phase a module is imported with is entirely driven from within ecma262 and JavaScript programs.
4. The ESM integration proposal specs the behavior for both source and evaluation phases.
5. Chrome isn't working on evaluation time Wasm module support at this time.
6. I haven't heard any commitment from Safari or Firefox for evaluation time Wasm module support.

Given the points (5) and (6) above, I'd prefer the informative note in this section of the wasm spec to be more reflective of near- (maybe medium-?) term web reality, but that's nothing to do with this PR.

[guybedford]

@syg thanks for your summary here, that sounds like a good outline to me.

With regards to:

Given the points (5) and (6) above, I'd prefer the informative note in this section of the wasm spec to be more reflective of near- (maybe medium-?) term web reality, but that's nothing to do with this PR.

I've added a possible adjustment to this note in https://github.com/WebAssembly/esm-integration/pull/91, and feel free to comment there further.

Just waiting for a final review from @nicolo-ribaudo then we should have all reviews completed for this to land.

[guybedford]

@domenic nits all look good to me. I do have a slight preference for Module Record over Cyclic Module Record terminology, but being clear works just as good. I've updated the top-line of the PR comment to include the answers to your questions, let me know if that answers things for you?

If there's a better way to handle the Phase 3 status of the ESM Integration in references, I'm open to alternatives.

[domenic]

If there's a better way to handle the Phase 3 status of the ESM Integration in references, I'm open to alternatives.

Could you give the document a different title, instead of having two documents with the same title?

[guybedford]

Sure, I've pushed a commit to update the ESM integration reference to be the spec repo instead of the spec document, and set the title as the direct proposal's name.

[domenic]

Sorry, that wasn't what I meant; I meant, can you update the spec document? I'm just very surprised the WASM CG is putting out two specs with the same name but different contents, and am unsure whether we should be referencing WASM CG documents if that's their status.

Edit: to be clear, I mean, can you update this line: https://github.com/WebAssembly/esm-integration/blob/8dbbb4dd19b479299cbe2f501a8b5ced9b61fbe8/document/js-api/index.bs#L2

[guybedford]

Okay, reverted the last commit and it seems the exceptions proposal did similar here previously in altering the title to include the proposal name. I've landed the title update in https://github.com/WebAssembly/esm-integration/pull/93, but still need to get the spec document to update properly through CI, so will fix that tomorrow and ping back here then. Let me know if there's anything else, and thanks for the reviews!

[guybedford]

@domenic the title on the spec is now updated, so we should finally be there now.

[nicolo-ribaudo]

I don't see a comment answering

Can you summarize why we relocated all the dependency checking to HostResolveImportedModule? That's a large change.

The summary is that we can split module loading in two phases:

1. Load a module
2. Load its dependencies

The first one ends at the end of the "create a X module" steps, and the second one is in HostLoadImportedModule.

Somewhere between 1 and 2 we need to validate those dependencies. Whether it was at the end of 1 or at the beginning of 2 was originally not observable. With source phase imports it is observable, because we can stop after 1, and we only want the validation to happen when we know that 2 is going to happen.