Support host exports besides function

[codefromthecrypt]

I started porting https://github.com/summerwind/the-art-of-webassembly-go by @summerwind to the latest API and realized we need to finish APIs exportable from the host, including memory, table and globals. Currently, we only support host functions.

[codefromthecrypt]

implementing this would be best done via a HostModule type because we currently enforce host modules once setup are immutable. This means we can't later add other things to them (ex globals)

[codefromthecrypt]

step 1: https://github.com/tetratelabs/wazero/pull/280

[mathetake]

let me take this as a part of #305

[codefromthecrypt]

let me take this as a part of https://github.com/tetratelabs/wazero/pull/305

please don't. the public API deserves to be designed carefully and not a side-car to figuring out internal details. There's a lot of coherence easy to break. There's literally no reason to do public api work to figure out which internal types should be exposed. Even which internal types are exposed is a very insignificant detail because they are internal so inaccessible by user code by definition.

[lthibault]

Would this allow me to make arbitrary Go types (like an interface) available to guest programs? I've been trying to figure out how to do so for the better part of a week 😄

Relatedly, would any of you be able to provide a bit more context on how this might work, and what would be involved? I'd really appreciate it, as I'm trying to familiarize myself with WASM internals as I venture into Wazero.

As I understand it, the mechanism through which a guest program written in Go interacts with host objects is through the syscall/js package, and as such, implementing this feature would involve mapping native Go structs to js.Values. Is that right? Any noteworthy details here?

Thanks!

[codefromthecrypt]

@lthibault so there are a couple ways.

• one way is to use a data binding approach such as https://github.com/knqyf263/go-plugin or https://karmem.org.
• another is to define host functions that access certain parts of the type in question.
  • implicit instance ex on the host it is in the go context
    • guest sig - (func $get_field (param $name) (param $name_len))
  • explicit instance ex the host has a lookup table map[uint32]*thing{}
    • guest sig - (func $get_field (param $this i32) (param $name) (param $name_len))

The implicit case is how we deal with http request/response (implicit as middleware is sync) https://github.com/http-wasm/http-wasm-guest-tinygo

In the explicit case, the guest could get the this (some call it context ID) param from propagating it everywhere, or from memory or another way. Note: Commonly, people use uint32 if mapping, though sometimes people use as uint64. If you use uint64, resist passing pointers directly as the guest is untrusted and could do arithmetic on the number.

There's yet another way, to use "reference types", which allows you to pass a host ref (like uintptr) on the stack, but the problem is at the moment no compiler supports it #532. We can support it when a compiler does.

does this help?

[codefromthecrypt]

@lthibault PS the current best way to propagate extra stuff is via go context. we do it all the time now. I added an explainer here, but we probably need an example for it https://github.com/tetratelabs/wazero/pull/826#issuecomment-1286454196

[lthibault]

@codefromthecrypt Is it possible to rely on syscall/js functions to pass arguments to host functions? I'm trying to make a complex data type available to the guest, whose method arguments are quite richly typed. If there were a way for the guest process to

1. instantiate a JS object
2. populate it with various fields (each being some JS, like Array, Object, Bool, etc.)
3. pass the object to the host-exported function

... that would make my life a fair bit easier!

Second question: if indeed this is possible, is this something most WASM compilers support?

[codefromthecrypt]

oh sorry my last reply was in general, not specific to GOOS=js. I'll update the PR description so I don't make that mistake again

[codefromthecrypt]

@lthibault

Is it possible to rely on syscall/js functions to pass arguments to host functions? I'm trying to make a complex data type available to the guest, whose method arguments are quite richly typed. If there were a way for the guest process to instantiate a JS object populate it with various fields (each being some JS, like Array, Object, Bool, etc.) pass the object to the host-exported function ... that would make my life a fair bit easier!

I re-titled this PR towards that, though the effort level is fairly high vs WebAssembly functions.

Second question: if indeed this is possible, is this something most WASM compilers support?

not really. I mean we barely support GOOS=js and it isn't 100pct compatible between implementations. https://wazero.io/languages/

There's an alternative which is to use the component-model (%.wit) IDL and that works more naturally though is drifty. There's also its predecessor (%.witx). %.wit doesn't rely on the component-model at runtime, for the moment anyway, so you may be able to define types this way. https://github.com/bytecodealliance/wit-bindgen

Another thing is you could ask if @inkeliz Karmem has plans to add JS bindings for the host-side of the generator..

[lthibault]

That's fair enough. I'm still getting a lay of the land, so I'm very open to recommendations (including "don't do that") 🙂 I think I'll dig more deeply into the context-based approach. My main issue there is how to pass complex datatypes in and out of the host function, but maybe this extra difficulty is just the price to pay for WASM.

[codefromthecrypt]

@lthibault yeah I think the mythical way is via externref mythical because nothing supports it. In lieu of that in wazero context is the dirty little secret which works fine to propagate any state (as long as the guest doesn't need to read it in wasm).

[codefromthecrypt]

this isn't going to happen as we are only maintaining GOOS=js until GOOS=wasip1 is out for two releases.