wazero

wazero copied to clipboard

Is your feature request related to a problem? Please describe.

I'm interested in implementing custom host functions defined via WITX in wazero, which can be accessed by a Wasm guest.

The host function WITX files for this project are from the Fastly Compute@Edge ABI defined here. This is currently implemented on a Rust runtime and it would be great to have this implemented on the wazero zero dependency Go runtime as well.

• compute-at-edge.witx
• typenames.witx

Describe the solution you'd like

The ability to implement a WITX defined host call in a wazero host implementation, ideally with an example.

Describe alternatives you've considered

I've looked at the counter example here. It would be nice to see how this maps to a WITX definition.

https://github.com/tetratelabs/wazero/blob/main/examples/namespace/counter.go

I'm unsure if this can be implemented given the lack of support for user-defined types.

https://github.com/tetratelabs/wazero/issues/420

Additional context

None at the moment.

quick context for starters.

• Host function signatures map directly to wasm signatures, so first step is to get a format to map to that (ex via %.wat)
• As far as I know, the witx format is deprecated for another format. ex https://github.com/WebAssembly/wasi-nn/pull/17
• I don't think user-defined types is the concern here as the same issue occurs in wasi even if they switched off witx mostly.

What's ultimately needed is code generator support in something like https://github.com/jedisct1/witx-codegen See if you can find one that can emit imports in a way convertible to %.wat is maybe the best start?

In other words it is probably a good first step to port a tool that already generates code from witx to generate go (specifically wazero host functions). That reduces the surface area to conversions and also buys time to figure out if witx is really deprecated or only deprecated in the WebAssembly orgs..

@aturon Do you know any plans for fastly and witx vs its successor? Also, it would be nice to be able to support this API even regardless of what might be migrating in terms of ABI formats. If you know something that can dump the ABI as %.wat imports I can help guide a manual impl meanwhile.

Thanks for the reply @codefromthecrypt! I've been looking WITX-Codegen. It doesn't support Go yet but once the conversion is well-defined, it could be enhanced to add Go code gen as it already handles WITX parsing. I haven't found a code gen or manual WITX implementation yet, so I believe one still needs to be written/understood for it to be implemented by a tool like WITX-Codegen. In a related area, I've done some work with a code gen tool for REST APIs, OpenAPI Generator, and would follow a process like that to add a new language.

If it helps on the manual side, we had to manually do wasi which is defined in witx https://github.com/WebAssembly/WASI/tree/snapshot-01/phases/snapshot/witx

We made some notes here https://github.com/tetratelabs/wazero/blob/main/wasi_snapshot_preview1/wasi.go#L433-L460 but generally wasi was easier because you could also look at a %.wat file of which there are numerous copies.

Another way is to look at something else which has attempted similar and see if that helps connect the dots. I would avoid trying to copy too much, rather use it for pointers (no pun intended). https://github.com/avidal/fastlike/blob/1d14d3f4767473eedf4a77e2a8904ffdd2f66e41/wasmcontext.go

I'll make time over the next week to make a sample thing for this maybe in a personal repo. then we can decide what next later (ex if @grokify or fastly want to own this)

@codefromthecrypt I've been chatting with the team at Fastly and it may make sense to wait a bit before digging into this. I am very excited about the idea of having the Fastly ABI implemented in wazero so I'm hoping we can do this soon.

@codefromthecrypt The Fastly team would love to help and see this happen. Please reach out to @dgryski on the Gophers Slack channel.

will do. I was actually cleaning house on our existing built-in imports in preparation, so you read my mind!

me thinking this is really exciting

There is also this Witz alternative with multi language support.

https://github.com/inkeliz/karmem#motivation

I used it with Wazero for a prototype for a test project. It’s very easy to use and very fast - see benchmarks.

Hi, I just tried to use the new wit format (WebAssembly Component Model). I think people will stick with witx for a while because it is quite opinionated. There are a lot of things, but the one most blaring is you cannot define functions with lower_snake_case.

Ex. you have to define functions literally as lower-hyphen or it breaks.

$ wit-bindgen host wasmtime-rust -e thing/thing.wit.md
Error: invalid character in identifier '_'
     --> thing/thing.wit.md:120:1
      |
  120 | get_thing: func(
      | ^

This doesn't match existing practice, so until some of these migration/flexibility issues are addressed, it is likely folks will stay on witx or ad-hoc ABI markups. Personally, I think defining functions as %.wat imports is most precise regardless of if there's an additional option for generation.

Ah interesting !! Thanks for the heads up. Send to be the way going forward. Will try it out !!

I am trying to run the same golang on the server and the browser.

Saw something about a new “unknown” triple target in tinygo which I think is trying to make the wasm have no std lib conceptually ?

This seems to be so that the tinygo wasm and wasi compiler conforms to this new WASM Component Model.

Am curious what you think of this for Browser side tool ?

https://github.com/inkeliz/go_inkwasm

it’s sort of doing the same thing to bridge and boot the the wasm in the browser.

the examples are amazing :

https://github.com/inkeliz/go_inkwasm/network/dependents

in terms of Host functions for the Browser, here is a library that gives you a FS and DB host side:

https://github.com/hack-pad/hackpadfs

It’s very close to working with tinygo and inside a Web Worker.

In the end the Host / Guest plumbing bus will match what Wazero offers , so you can have many wasm workers running in Wazero or a browser is my aim.

NATS acting as a bus to allow all these “ wasm workers “ to talk to each other

Saw something about a new “unknown” triple target in tinygo which I think is trying to make the wasm have no std lib conceptually. This seems to be so that the tinygo wasm and wasi compiler conforms to this new WASM Component Model ?

I don't think these are related. At least I haven't seen anything directly related between the freestanding target and component model. Indirectly, if you have something freestanding you at least are decoupled from existing wasi which could make an incompatible new wasi possible to integrate. OTOH to integrate that is no longer freestanding ;)

wit parser and generator in golang

https://github.com/alecthomas/wit-go

the underlying golang libs are really nicely done too .

example golang project that uses it to gen golang code to interface with a rust wasm system

https://github.com/alecthomas/lunatic-go

What ya think ??

@codefromthecrypt @grokify