wazero support compile python?

[mougeCM]

package main

import (
	"context"
	"embed"
	_ "embed"
	"io/fs"
	"log"

	"github.com/tetratelabs/wazero"
	"github.com/tetratelabs/wazero/wasi_snapshot_preview1"
)

//go:embed testdata/env.py
var envPy embed.FS

//go:embed testdata/bin/python3.wasm
var greetWasm []byte

func main() {
	ctx := context.Background()

	// 新建一个 wasm 运行时
	r := wazero.NewRuntimeWithConfig(ctx, wazero.NewRuntimeConfig().
		WithWasmCore2())
	defer r.Close(ctx)

	// 实例化一个名为“env”的 Go 定义的模块，该模块导出一个函数以记录到控制台
	_, err := r.NewModuleBuilder("env").
		Instantiate(ctx, r)
	if err != nil {
		log.Panicln(err)
	}

	if _, err = wasi_snapshot_preview1.Instantiate(ctx, r); err != nil {
		log.Panicln(err)
	}

	code, err := r.CompileModule(ctx, greetWasm, wazero.NewCompileConfig())
	if err != nil {
		log.Panicln(err)
	}

	rooted, err := fs.Sub(envPy, "testdata")
	if err != nil {
		log.Panicln(err)
	}

	config := wazero.NewModuleConfig().WithFS(rooted)

	env := config.
		WithEnv("PYTHONHOME", "testdata/lib/python3.11").
		WithEnv("PYTHONPATH", "testdata/lib/python3.11")
	if _, err = r.InstantiateModule(ctx, code, env); err != nil {
		log.Panicln(err)
	}
}

I Already have a python3.wasm，I want to run the env.py on the python3.wasm

Use wasmer run testdata/bin/python3.wasm --mapdir /::./ --env 'PYTHONHOME=testdata/lib/python3.11' --env 'PYTHONPATH=testdata/lib/python3.11' -- testdata/env.py command is ok

But use wazero will appear error

2022/08/26 16:20:41 module "" closed with exit_code(1)
panic: module "" closed with exit_code(1)


goroutine 1 [running]:
log.Panicln({0xc001255e40?, 0x11b2bd0?, 0xc0000000d8?})
        /usr/local/go/src/log/log.go:399 +0x65
main.main()
        /Users/bytedance/goProject/src/code.byted.org/chenmin.tate/wasm_project/python/greet.go:57 +0x50f
exit status 2

May be I have a problem of the usage，ask for help～

[mougeCM]

I tested it and the mapdir is missing

[mougeCM]

wasmer has the MapDirectory is similar to the wasmer-go?

wasmBytes, _ := ioutil.ReadFile("module.wasm")

	store := wasmer.NewStore(wasmer.NewEngine())
	module, _ := wasmer.NewModule(store, wasmBytes)

	wasiEnv, _ := wasmer.NewWasiStateBuilder("wasi-program").
		// Choose according to your actual situation
		// Argument("--foo").
		// Environment("ABC", "DEF").
		// MapDirectory("./", ".").
		Finalize()

[codefromthecrypt]

The way to grant access to files is via WithFS and you can only give one fs.FS.

Do you mind making a GitHub repo with this project or something similar in it (using the same wasm), as it will be easier to identify what's going on. Particularly, you can use the experimental logging filter to see where it crashes.

If you open that repo, you can ping me on an issue there and I can help you troubleshoot over the next couple days (free time). this will reduce the amount of spam here, then if we find a solution we can report back what it was here.

[mougeCM]

Is the current version ready for production?

My main requirement is: wasm and go communicate with each other through linear memory

[codefromthecrypt]

@mougeCM wazero will have a beta release end of this month and 1.0 is scheduled for early next year. We don't test python yet so I wouldn't recommend thinking it is ready for production. This is also why I asked you if you can create a demo repo so I can help you figure out where the problem exists. For example, some language compilers are better for troubleshooting, for example, having helpful error messages to tell if some problem is due to an unimplemented syscall in wazero, or some undefined behavior or something else. You are the first to ask about python, so until at least your demo works and you can know why it does or doesn't in the context of python I wouldn't recommend using it :D

[codefromthecrypt]

Also it seems python and wasm is very much a work in progress. There are two different ways that wasm might have been built, ex via emscripten or wasi. Without knowing the source and how the wasm binary was compiled, we can't guess what's going on. Can you mention how you downloaded or compiled python3.wasm? This is an example of what can be in the troubleshooting repo

https://pythondev.readthedocs.io/wasm.html#wasi

[codefromthecrypt]

Taking a guess that this is using what seems to be the de'facto python build from https://github.com/singlestore-labs/python-wasi (25MB of python3.11.wasm), I would say it is likely that an error is due to less syscalls (WASI) supported in wazero vs what it imports.

Since it is a dynamic language it is hard to tell which you are hitting. It also remains curious what the error is. Anyway, once you confirm what python you are using, we can proceed to inventory, but I suspect regardless, the closest thing we could support would be the more dominant wasi implementation.

https://wazero.io/specs/#webassembly-system-interface-wasi

$ wasm2wat python3.11.wasm|grep '__imported_wasi_snapshot_preview1_'|grep -v '(import'|sort|uniq
    call $__imported_wasi_snapshot_preview1_args_get
    call $__imported_wasi_snapshot_preview1_args_sizes_get
    call $__imported_wasi_snapshot_preview1_clock_res_get
    call $__imported_wasi_snapshot_preview1_clock_time_get
    call $__imported_wasi_snapshot_preview1_environ_get
    call $__imported_wasi_snapshot_preview1_environ_sizes_get
    call $__imported_wasi_snapshot_preview1_fd_advise
    call $__imported_wasi_snapshot_preview1_fd_allocate
    call $__imported_wasi_snapshot_preview1_fd_close
    call $__imported_wasi_snapshot_preview1_fd_datasync
    call $__imported_wasi_snapshot_preview1_fd_fdstat_get
    call $__imported_wasi_snapshot_preview1_fd_fdstat_set_flags
    call $__imported_wasi_snapshot_preview1_fd_filestat_get
    call $__imported_wasi_snapshot_preview1_fd_filestat_set_size
    call $__imported_wasi_snapshot_preview1_fd_filestat_set_times
    call $__imported_wasi_snapshot_preview1_fd_pread
    call $__imported_wasi_snapshot_preview1_fd_prestat_dir_name
    call $__imported_wasi_snapshot_preview1_fd_prestat_get
    call $__imported_wasi_snapshot_preview1_fd_pwrite
    call $__imported_wasi_snapshot_preview1_fd_read
    call $__imported_wasi_snapshot_preview1_fd_readdir
    call $__imported_wasi_snapshot_preview1_fd_seek
    call $__imported_wasi_snapshot_preview1_fd_sync
    call $__imported_wasi_snapshot_preview1_fd_tell
    call $__imported_wasi_snapshot_preview1_fd_write
    call $__imported_wasi_snapshot_preview1_path_create_directory
    call $__imported_wasi_snapshot_preview1_path_filestat_get
    call $__imported_wasi_snapshot_preview1_path_link
    call $__imported_wasi_snapshot_preview1_path_open
    call $__imported_wasi_snapshot_preview1_path_readlink
    call $__imported_wasi_snapshot_preview1_path_remove_directory
    call $__imported_wasi_snapshot_preview1_path_rename
    call $__imported_wasi_snapshot_preview1_path_symlink
    call $__imported_wasi_snapshot_preview1_path_unlink_file
    call $__imported_wasi_snapshot_preview1_poll_oneoff
    call $__imported_wasi_snapshot_preview1_proc_exit
    call $__imported_wasi_snapshot_preview1_random_get
    call $__imported_wasi_snapshot_preview1_sched_yield

[mougeCM]

1. Accessing python is my second job. My current first job is connecting to Go (Host: Go, SandBox: Go). Can the current beta support my production needs?

[mougeCM]

Runing Python in WebAssembly: https://www.fermyon.com/blog/python-wagi python3.wasm from: https://github.com/fermyon/wagi-python/tree/main/opt/wasi-python/bin

[inkeliz]

What is "connecting"? If that is "wasm and go communicate with each other through linear memory": you can use Memory() (https://github.com/tetratelabs/wazero/blob/06ad22f33fc4/api/wasm.go#L140-L141) to read/write data direct into the memory, without any WASI requirement. Of course, the "memory" contains the wasm controlled information, so you can't write at random places. You need to create some slice/array on the guest-side (beware of GC!), then give the pointer to the host (calling exported function, or exporting a function to be called), then you can write that into that specific index, using something like: Memory().Write(context.Background(), uint32(index_from_guest), []byte{....}).

I'm currently using Wazero with 7 different languages (TinyGo, Zig, AssemblyScript, Odin, .NET, Swiftwasm, C), and I'm writing/reading that using the Memory(), and calling exported functions to get the indexes of the array. So, Wazero is capable to do that, and already have examples exchanging data.

If you want to "connect" by calling some WASI Filesystem, then that is another history. I never used that feature, but I think Wazero supports that too.

[codefromthecrypt]

Accessing python is my second job. My current first job is connecting to Go (Host: Go, SandBox: Go). Can the current beta support my production needs?

@mougeCM In summary, you can use wazero now, but anything you use in production should be tested by you. Until wazero turns 1.0 in Feb there's a chance of API changes, but we hope to not have too many breaks.

The beta that will be released in the next couple days is already in use, ex by Trivy, anyone using waPC as well projects GitHub knows about and those it doesn't.

Host: Go, SandBox: Go is the most common setup, if by Go you mean TinyGo. If you mean normal Go as in GOARCH=wasm GOOS=js, it is not often used.

I would recommend reading https://wazero.io/languages/ on the language choice you feel dominant. For support, wazero isn't well placed to support the guest side deep on every language, that's why we have links to their support. Support includes the libraries you use also, and ones that are tested with wazero, like https://github.com/inkeliz/karmem, are likely to have a better experience.

Meanwhile, this is an issue and typically we use issues for things like code changes, not open questions. Please watch this repo for updates as when we release our beta we'll mention a chat channel for use.

[robbertvanginkel]

I ran into this as well when attempting to run CPython 3.11's new wasm target in wazero (https://docs.python.org/3/whatsnew/3.11.html#build-changes).

My reproduction steps:

$ git clone https://github.com/python/cpython.git
$ git checkout 3.11 # v3.11.0 release has issues, checkout patch branch for now
$ git rev-parse HEAD
2d5f4ba17480c1f883a0822c90af25d2ec9bf7ed
$ WASI_SDK_PATH=../wasi-sdk-16.0 ./Tools/wasm/wasm_build.py wasi
...
# Before building the stdlib, need to make sure it is not compressed
# That seems to work for the emscripten target, but not wasi.
$ git apply - <<-EOF
diff --git a/Tools/wasm/wasm_assets.py b/Tools/wasm/wasm_assets.py
index 6557e3f37a..7a1696d700 100755
--- a/Tools/wasm/wasm_assets.py
+++ b/Tools/wasm/wasm_assets.py
@@ -1,4 +1,4 @@
-#!/usr/bin/env python
+#!/usr/bin/env python3
 """Create a WASM asset bundle directory structure.

 The WASM asset bundles are pre-loaded by the final WASM build. The bundle
@@ -232,8 +232,8 @@ def main():

     # bpo-17004: zipimport supports only zlib compression.
     # Emscripten ZIP_STORED + -sLZ4=1 linker flags results in larger file.
-    args.compression = zipfile.ZIP_DEFLATED
-    args.compresslevel = 9
+    args.compression = zipfile.ZIP_STORED
+    args.compresslevel = 0

     args.builddir = get_builddir(args)
     args.sysconfig_data = get_sysconfigdata(args)
EOF
$ cd builddir/wasm
$ make wasm_stdlib
...

Where wasmtime can drop me into a repl

$ wasmtime --mapdir /usr/local/lib::usr/local/lib python.wasm       
Python 3.11.0+ (heads/3.11-dirty:2d5f4ba174, Dec  1 2022, 15:50:36) [Clang 14.0.4 (https://github.com/llvm/llvm-project 29f1039a7285a5c3a9c353d05414 on wasi
Type "help", "copyright", "credits" or "license" for more information.
>>> ^D

Whereas with wazero:

$ go install github.com/tetratelabs/wazero/cmd/wazero@latest
go: downloading github.com/tetratelabs/wazero v1.0.0-pre.4
$ wazero run --mount usr/local/lib:/usr/local/lib/ python.wasm
Fatal Python error: init_sys_streams: can't initialize sys standard streams
Python runtime state: core initialized
OSError: [Errno 8] Bad file descriptor

Current thread 0x00000000 (most recent call first):
  <no Python frame>

Using https://pkg.go.dev/github.com/tetratelabs/[email protected]/experimental#example-package-CustomListenerFactory, I added a small listener that would print every time def.ModuleName() == "wasi_snapshot_preview1", I can see at least the following thing marked as missing in https://wazero.io/specs/#webassembly-system-interface-wasi are called:

wasi_snapshot_preview1.fd_fdstat_get
wasi_snapshot_preview1.fd_readdir
wasi_snapshot_preview1.path_filestat_get
wasi_snapshot_preview1.path_readlink

$ wasm-objdump --section 'import' --details python.wasm

python.wasm:    file format wasm 0x1

Section Details:

Import[43]:
 - func[0] sig=1 <__imported_wasi_snapshot_preview1_args_get> <- wasi_snapshot_preview1.args_get
 - func[1] sig=1 <__imported_wasi_snapshot_preview1_args_sizes_get> <- wasi_snapshot_preview1.args_sizes_get
 - func[2] sig=1 <__imported_wasi_snapshot_preview1_environ_get> <- wasi_snapshot_preview1.environ_get
 - func[3] sig=1 <__imported_wasi_snapshot_preview1_environ_sizes_get> <- wasi_snapshot_preview1.environ_sizes_get
 - func[4] sig=1 <__imported_wasi_snapshot_preview1_clock_res_get> <- wasi_snapshot_preview1.clock_res_get
 - func[5] sig=23 <__imported_wasi_snapshot_preview1_clock_time_get> <- wasi_snapshot_preview1.clock_time_get
 - func[6] sig=24 <__imported_wasi_snapshot_preview1_fd_advise> <- wasi_snapshot_preview1.fd_advise
 - func[7] sig=25 <__imported_wasi_snapshot_preview1_fd_allocate> <- wasi_snapshot_preview1.fd_allocate
 - func[8] sig=0 <__imported_wasi_snapshot_preview1_fd_close> <- wasi_snapshot_preview1.fd_close
 - func[9] sig=0 <__imported_wasi_snapshot_preview1_fd_datasync> <- wasi_snapshot_preview1.fd_datasync
 - func[10] sig=1 <__imported_wasi_snapshot_preview1_fd_fdstat_get> <- wasi_snapshot_preview1.fd_fdstat_get
 - func[11] sig=1 <__imported_wasi_snapshot_preview1_fd_fdstat_set_flags> <- wasi_snapshot_preview1.fd_fdstat_set_flags
 - func[12] sig=1 <__imported_wasi_snapshot_preview1_fd_filestat_get> <- wasi_snapshot_preview1.fd_filestat_get
 - func[13] sig=26 <__imported_wasi_snapshot_preview1_fd_filestat_set_size> <- wasi_snapshot_preview1.fd_filestat_set_size
 - func[14] sig=24 <__imported_wasi_snapshot_preview1_fd_filestat_set_times> <- wasi_snapshot_preview1.fd_filestat_set_times
 - func[15] sig=27 <__imported_wasi_snapshot_preview1_fd_pread> <- wasi_snapshot_preview1.fd_pread
 - func[16] sig=1 <__imported_wasi_snapshot_preview1_fd_prestat_get> <- wasi_snapshot_preview1.fd_prestat_get
 - func[17] sig=3 <__imported_wasi_snapshot_preview1_fd_prestat_dir_name> <- wasi_snapshot_preview1.fd_prestat_dir_name
 - func[18] sig=27 <__imported_wasi_snapshot_preview1_fd_pwrite> <- wasi_snapshot_preview1.fd_pwrite
 - func[19] sig=4 <__imported_wasi_snapshot_preview1_fd_read> <- wasi_snapshot_preview1.fd_read
 - func[20] sig=27 <__imported_wasi_snapshot_preview1_fd_readdir> <- wasi_snapshot_preview1.fd_readdir
 - func[21] sig=28 <__imported_wasi_snapshot_preview1_fd_seek> <- wasi_snapshot_preview1.fd_seek
 - func[22] sig=0 <__imported_wasi_snapshot_preview1_fd_sync> <- wasi_snapshot_preview1.fd_sync
 - func[23] sig=1 <__imported_wasi_snapshot_preview1_fd_tell> <- wasi_snapshot_preview1.fd_tell
 - func[24] sig=4 <__imported_wasi_snapshot_preview1_fd_write> <- wasi_snapshot_preview1.fd_write
 - func[25] sig=3 <__imported_wasi_snapshot_preview1_path_create_directory> <- wasi_snapshot_preview1.path_create_directory
 - func[26] sig=8 <__imported_wasi_snapshot_preview1_path_filestat_get> <- wasi_snapshot_preview1.path_filestat_get
 - func[27] sig=29 <__imported_wasi_snapshot_preview1_path_filestat_set_times> <- wasi_snapshot_preview1.path_filestat_set_times
 - func[28] sig=17 <__imported_wasi_snapshot_preview1_path_link> <- wasi_snapshot_preview1.path_link
 - func[29] sig=30 <__imported_wasi_snapshot_preview1_path_open> <- wasi_snapshot_preview1.path_open
 - func[30] sig=16 <__imported_wasi_snapshot_preview1_path_readlink> <- wasi_snapshot_preview1.path_readlink
 - func[31] sig=3 <__imported_wasi_snapshot_preview1_path_remove_directory> <- wasi_snapshot_preview1.path_remove_directory
 - func[32] sig=16 <__imported_wasi_snapshot_preview1_path_rename> <- wasi_snapshot_preview1.path_rename
 - func[33] sig=8 <__imported_wasi_snapshot_preview1_path_symlink> <- wasi_snapshot_preview1.path_symlink
 - func[34] sig=3 <__imported_wasi_snapshot_preview1_path_unlink_file> <- wasi_snapshot_preview1.path_unlink_file
 - func[35] sig=4 <__imported_wasi_snapshot_preview1_poll_oneoff> <- wasi_snapshot_preview1.poll_oneoff
 - func[36] sig=7 <__imported_wasi_snapshot_preview1_proc_exit> <- wasi_snapshot_preview1.proc_exit
 - func[37] sig=2 <__imported_wasi_snapshot_preview1_sched_yield> <- wasi_snapshot_preview1.sched_yield
 - func[38] sig=1 <__imported_wasi_snapshot_preview1_random_get> <- wasi_snapshot_preview1.random_get
 - func[39] sig=3 <__imported_wasi_snapshot_preview1_sock_accept> <- wasi_snapshot_preview1.sock_accept
 - func[40] sig=16 <__imported_wasi_snapshot_preview1_sock_recv> <- wasi_snapshot_preview1.sock_recv
 - func[41] sig=8 <__imported_wasi_snapshot_preview1_sock_send> <- wasi_snapshot_preview1.sock_send
 - func[42] sig=1 <__imported_wasi_snapshot_preview1_sock_shutdown> <- wasi_snapshot_preview1.sock_shutdown 

While fd_fdstat_get is marked as implemented in https://wazero.io/specs/#webassembly-system-interface-wasi, the implementation seems incomplete: https://github.com/tetratelabs/wazero/blob/40e698e068d9daa98a1660b7a0a31a1886ec3691/imports/wasi_snapshot_preview1/fs.go#L171

I took a bit of a stab at implementing the stat & readdir related ones in imports/wasi_snapshot_preview1/fs.go, but I realized quickly that it's not as straightforward as I'd hoped. I could get something to work by making the fs.FS in sys.FSContext public, I figured that that this would be a good time to start asking for help.

Is this something you'd be open to guiding an external contributor through or may be easier done by people more familiar with wazero?

[robbertvanginkel]

Whoa, I now see that https://github.com/tetratelabs/wazero/pull/865, https://github.com/tetratelabs/wazero/pull/858 & https://github.com/tetratelabs/wazero/pull/838 all went in in the last few days! This is really great!

The CustomListenerFactory that I had attached was actually when I was on the v1.0.0-pre.3 release. I'll update and see if I can more concretely pinpoint what is missing!

[robbertvanginkel]

It seems like

• https://github.com/tetratelabs/wazero/blob/40e698e068d9daa98a1660b7a0a31a1886ec3691/imports/wasi_snapshot_preview1/fs.go#L171-L175
• https://github.com/tetratelabs/wazero/blob/40e698e068d9daa98a1660b7a0a31a1886ec3691/imports/wasi_snapshot_preview1/fs.go#L278-L279

Need to special case the fd numbers for stdin/stdout/stderr. Once I have those write stat results indicating they are a character device with read/write rights python recognizes them as tty and drops me into a wazero based repl. I would have to read up more on exactly how the rights work before I could make a PR, I may get to that later but wanted to share in case anyone else is more familiar.

[codefromthecrypt]

@robbertvanginkel excellent work! ps "rights" has been dropped from wasi and no one implements it, so you can ignore it. It is a goal to complete python and finally seems we have some action to take!

It would be grand if you are up to completing the impl, but either way this helps. Keep us posted here or on gophers slack #wazero-dev channel.

• I forget if you are already there (if not, you may need an invite to join gophers).

[codefromthecrypt]

I'll implement fdstat today, as I ran into it for another reason

[codefromthecrypt]

@robbertvanginkel please against https://github.com/tetratelabs/wazero/pull/895 or after merge!

[codefromthecrypt]

@robbertvanginkel also, if you get a chance, can you check if latest cpython still needs hacks for wasi to build correctly? at some point would be nice if some version of python+wasi works and can be published somewhere without instructions

[robbertvanginkel]

The repl works now!

$ go install github.com/tetratelabs/wazero/cmd/wazero@main
$ wazero run --mount usr/local/lib:/usr/local/lib/ python.wasm
Python 3.11.0+ (heads/3.11-dirty:2d5f4ba174, Dec  7 2022, 11:12:35) [Clang 14.0.4 (https://github.com/llvm/llvm-project 29f1039a7285a5c3a9c353d05414 on wasi
Type "help", "copyright", "credits" or "license" for more information.
>>> 1 + 1 
2
>>> 

In addition to fixing python, it actually also fixed a stat thing I was running into when using SwiftWasm so double thanks 🎉

Python does sort of work without patches, on a fresh build from the cpython repo this does work:

$ wazero run -env PYTHONPATH=/builddir/wasi/build/lib.wasi-wasm32-3.11:/Lib -mount /Users/robbert/dev/cpython:/ builddir/wasi/python.wasm
Python 3.11.0+ (heads/3.11-dirty:2d5f4ba174, Dec  7 2022, 11:30:37) [Clang 14.0.4 (https://github.com/llvm/llvm-project 29f1039a7285a5c3a9c353d05414 on wasi
Type "help", "copyright", "credits" or "license" for more information.
>>> 

This works for development, but bundling a stdlib makes more sense for a deployment. zlib support for the wasi build is tracked in https://github.com/python/cpython/issues/93819, once that's fixed the output of make wasm_stdlib should work without modifications.

[codefromthecrypt]

@robbertvanginkel you are a champ. Let's leave this open until the last glitches are out or timeout. Once make wasm_stdlib works, let's close this.

[codefromthecrypt]

https://github.com/python/cpython/issues/93819 doesn't seem to be going anywhere, so I raised this PR instead to close out the issue. Thanks all, for the attention! https://github.com/fermyon/wagi-python/pull/4

[codefromthecrypt]

updates:

• there seems more maintenance upstream despite the zlib thing outstanding https://github.com/python/cpython/pull/101003
• there's another build curated by vmware, though I've not tried it with wazero https://github.com/vmware-labs/webassembly-language-runtimes/tree/main/python

@mougeCM @robbertvanginkel if either of you have time, I'm curious what use cases you have, or have heard of, for an embedded python interpreter.

Is it for the repl, or for a script? Are the interesting parts sandboxing, or embedding or something else? Have you needed or tried packages or otherwise custom libraries?

I'm nervous about saying game over because we can start the repl. In zig and tinygo, we currently run the stdlib OS tests which raise confidence that most things work (or are known to not work due to things outside our control, such as syscalls unmapped to wasi, etc). I wonder how we can achieve confidence python works.

Anyway, any thoughts appreciated!

[codefromthecrypt]

wrt https://github.com/vmware-labs/webassembly-language-runtimes/tree/main/python

I personally can't get the non-wasmedge variant working, as it errs like below. Their docker image is also wasmedge-specific which clashes with signatures in normal wasi. For now, best to use fermyon until those things sort out.

Fatal Python error: init_fs_encoding: failed to get the Python codec of the filesystem encoding
Python runtime state: core initialized
ModuleNotFoundError: No module named 'encodings'

[robbertvanginkel]

Our python usecase is to run a script as a plugin to our protobuf compiler.

The high level is that protobuf is a language independent schema format, in which most language bindings are generated from plug-in binaries that reads an encoded form of the schema from stdin and writes generated code as an encoded message to stdout. There's an existing ecosystem in out there that we'd like to support where instead of looking up native binaries from a user's PATH, we allow them to specify a single wasm file and run that instead.

For python (and other interpreted languages like js) we currently compile the interpreter to a wasm file, bundle the stdlib and plugin code (packaged as a pyz with zipapp) in a custom wasm section. We read that out and make it available to wazero before starting the run. Using the build script I provided to the python team at https://github.com/python/cpython/issues/93819#issuecomment-1384541880 to compile with zlib support, this works well for us!

[mathetake]

curious what's left to close this?

[codefromthecrypt]

It was blocking on https://github.com/python/cpython/issues/93819, but really this is incidental to wazero. We can close it and someone can know that once ^^ is resolved the normal cpython build can work instead of relying on community repackaging