Worker imports & memory sharing

[guybedford]

As brought up in https://github.com/WebAssembly/esm-integration/issues/14#issuecomment-2931717336, one problem with the ESM integration instance import model is that for multi-threading scenarios the memory import needs to be available at startup.

The ESM Phase Imports proposal also enables the ability for new Worker(wasmModule) to create a worker from a WebAssembly module which is currently specified in https://github.com/WebAssembly/esm-integration/pull/106.

Delaying instantiation via dynamic import provides one route, but it might be worth considering the ergonomics more carefully here.

One idea might be to allow passing module imports to the new Worker() constructor when using the new source form:

import source mod from './mod.wasm';
const memory = new WebAssembly.Memory(...);
const worker = new Worker(source, {
  imports: {
    'memory': memory
  }
});

Normally for dynamic import() this sort of resolution override is dangerous due to mismatch with existing imports, but as the creator of the module environment it could be framed as a sort of top-level-only import map, so that setting imports for context creation can certainly work actually.

@sbc100 let me know if I'm summarizing the issue correctly or missing anything here too.

[sbc100]

For emscripten and the web (at least for now) we always a JS file to start the worker, and that JS file then loads the wasm file.

However, because we currently have to wait for memory to be received via postMessage we currently fall back to a dynamic import. See https://github.com/emscripten-core/emscripten/pull/24555. And specially the worker helper code that does the async load once the memory is available: https://github.com/emscripten-core/emscripten/blob/main/src/pthread_esm_startup.mjs.

When building this solution I had though about suggesting additional argument to new Worker, essentially allowing the same arguments that postMessage takes, so you could do something like const worker = new Worker('prog.js', initialMessage) where initialMessage would contain the wasm memory and would be globally available in the worker global scope somewhere, allowing the message contents to be available during the ESM instantiation phase.

[guybedford]

Hmm, perhaps imports here could apply equally to JS module workers somehow?

new Worker('prog.js', {
  "type": "module",
  "imports": {
    "memory": memory
  }
});

@nicolo-ribaudo already worked on a spec for passing an importMap previously in https://github.com/whatwg/html/pull/10858. Although this is more complex in allowing a sort of synthetic module to be defined in that import map with some transferred value, so not sure if there's an easy way to simplify that.

[sbc100]

I think that would work great. (How exactly would prog.js access the memory import in that case?)

[guybedford]

I guess it could be the same pattern as the WebAssembly instantiation API here in being double keyed by the module specifier and module export name:

new Worker('prog.js', {
  "type": "module",
  "imports": {
    "memory": {
      default: memory
    }
  }
});

Where the above implies that the values are postMessage'd and a SyntheticModuleRecord constructed on the other side. As I say the exception from dynamic imports not being able to support this pattern that makes such a thing possible in this case is that it's not possible to race the top-level module when creating the entire context, making it the first example in JS modules where this instantiation model could be supported without creating module keying issues.

So I think such a feature likely is orthogonal to the import map after all... it would just effectively be a variation on initialMessage though.

[tlively]

Is initialMessage an existing feature I can read about somewhere?

I like the idea of being able to supply additional arguments to be postMessaged to the worker on creation, but I don't understand the answer to @sbc100's question about how the worker JS would actually receive those arguments.

[guybedford]

This is something supported via workerData for Node.js worker threads, which then becomes available as an import.

So the discussion was around if a similar special import could be used to provide this initial message if HTML were to support something like it because of the shared memory import use case to allow the module to still effecitvely be statically initialized.

[Vados80]

This is something supported via workerData for Node.js worker threads, which then becomes available as an import.

So the discussion was around if a similar special import could be used to provide this initial message if HTML were to support something like it because of the shared memory import use case to allow the module to still effecitvely be statically initialized.

module.exports

[sbc100]

Something like nodejs' workerData could be perfect yes. In fact I can try to prototype something that runs under node using that method to confirm that it is sufficient.

[guybedford]

I've posted an upstream HTML issue for an initData worker initialization in https://github.com/whatwg/html/issues/11749. This is slightly different to the concept discussed here around import map init data. If there is implementer interest, follow-ups in that thread would be appreciated!