pypa
Support for wasm32-emscripten wheels
@henryiii has discussed adding a wasm32-emscripten target to cibuildwheel in the past. I would like to help with this. Currently it is not super invasive to build such wheels.
Here's numpy's CI support for Emscripten using pyodide build:
https://github.com/numpy/numpy/blob/main/.github/workflows/emscripten.yml
Rust + Python projects can directly use maturin: https://github.com/pydantic/pydantic-core/blob/main/Makefile#L46
There's currently not all that many examples of projects building against Emscripten in their CI, but I would like to change that!. I think a lot of maintainers are enthusiastic about the concept.
I'm keen for this too. WASM is the next universal VM architecture! Looks like pyodide have done a lot of work to make this easier.
Why do we target wasm32 - a 32bit architecture? Is that the most common?
We don't want to depend on the setup-emsdk action, so we'd have to roll that functionality in. Luckily, that's pretty simple.
We'd also have to decide if it should be enabled by default, as I don't think PyPI is accepting these wheels yet.
I think it should be a platform, so it would never being enabled by default, you’d have to ask for that platform. Currently, wheels have to target a specific version, there’s no backward compatibility guarantee, so they can’t be uploaded to PyPI; they aren’t really redistributable. But I think that’s fine for an opt-in platform, we could just have a warning for now.
I was originally thinking this should run on docker, but looks like maybe that’s not needed?
Why do we target wasm32 - a 32bit architecture? Is that the most common?
Well:
- wasm64 doesn't really exist yet (it's specced but not implemented)
- 32 bit pointers make code size much smaller
- wasm is a harvard architecture -- function pointers and data pointers live in different address spaces. This reduces the amount of stuff in the address space by a fair amount.
So far very few people are running into the 4 gb limit.
I notice that build-frontend defaults to pip. We can't use pip as the build frontend for Pyodide at this point. Can I just print a warning and ignore this setting?
I suppose technically the build-frontend is pyodide? yes feel free to ignore this for now.
I was originally thinking this should run on docker, but looks like maybe that’s not needed?
Let's see how it goes - I think I'd slightly rather it was not on docker, but if that turns out to be a really easy way to get a reproducible environment I wouldn't be against it either.
Successfully built and tested multidict with
cibuildwheel --platform pyodide
I'm trying to build numpy now but it pins wheel==0.38.1 in it's build requires whereas cibuildwheel/resources/constraints-python310.txt has wheel==0.40.0. Am I using the cibuildwheel constraints file wrong?
It looks like numpy pins cibuildwheel==2.12.0 which may have something to do with the wheel version...
Yes, environments are pinned by cibuildwheel version (unless you request otherwise, via https://cibuildwheel.readthedocs.io/en/stable/options/#dependency-versions )
How is numpy pinning cibuildwheel if you are building it?
As for the build frontend, I'd say ideally make it a warning or even an error if a user specifically sets it to pip, but no warning/error if it's unset, if possible. Not sure when we think we might want to change the default.
Numpy pins the version here: https://github.com/numpy/numpy/blob/main/tools/ci/cirrus_general.yml#L3 I get a crash because of the wheel version conflict. I removed one of the wheel versions and now I successfully build the wheel. Working on getting the numpy tests to run...