Pyodide Wheels

[DaAwesomeP]

Description

I'm not sure if this is out of the scope of this project, but the ability to build Pyodide wheels would be wonderful: https://pyodide.org/en/stable/development/new-packages.html

Build log

No response

CI config

No response

[henryiii]

The numpy.org webpage now has a pyodide-powered scientific stack in the browser. Wow.

I have no idea ATM what is involved with building a pyodide wheel, and I'd assume there really aren't "platforms" for them, and I think a lot would likely be needed upstream to standardize tags and such, and cibuildwheel likely might not be the place to produce them, but I really have no idea. I don't even know if they are in traditional files. If it was something cibuildwheel could do, it would probably be a new "platform".

[henryiii]

You can see this mentioned in the roadmap here: https://pyodide.org/en/stable/project/roadmap.html

https://github.com/pyodide/pyodide/pull/655 & https://github.com/pyodide/pyodide/pull/2027 are linked there. Both of which seem to be merged.

[DaAwesomeP]

The numpy.org webpage now has a pyodide-powered scientific stack in the browser. Wow.

I know, it's pretty cool!

Both of which seem to be merged.

Yeah, I was under the impression that they are already using wheels. This documentation seems to hint that you can create wheels: https://pyodide.org/en/stable/development/new-packages.html

[henryiii]

Yes, they do make wheels now. Looking into the process. I just found that page, actually. :)

[hoodmane]

For the platform compatibility question see https://github.com/emscripten-core/emscripten/issues/15917

[henryiii]

Pretty sure there’s no way to load the a non pure wheel from anything but pyodide’s own build process at the moment. I was able to compile for it though locally (took a couple of hours to build CPython and all)

https://github.com/pyodide/pyodide/discussions/2167

[hoodmane]

If you build a compatible non-pure wheel in some other way, it's possible to load it with pyodide.loadPackage(url).

[DaAwesomeP]

Maybe more feasible and makes more sense to implement, this is now beginning to be added as an official architecture in CPython: https://github.com/python/cpython/blob/main/Tools/wasm/README.md

[rth]

So @henryiii do you have any suggestions of how we can start working this? Thanks to @hoodmane work out-of-tree builds are starting to looks reasonably good (example in https://github.com/numpy/numpy/pull/21895 where the test runner is also going to be shipped by Pyodide soon). So how do we move forward on this?

From what I can tell for simple packages, right now the constraints are,

• at least in the numpy CI the build seems to be fine in GHA with ubuntu-latest, but I wonder whether it wouldn't be better to build in the pyodide-env Docker container anyway, similarly to manylinux.
• running tests is still a bit WIP so there should be a way to disable it just for this build (particularly, that there is bound to be some tests that need fixing for WASM)
• whatever we do, it should still be a somewhat experimental opt-in feature and not enabled by default.
• and also we still have nowhere to upload the produced wheels https://github.com/pyodide/pyodide/issues/3049 short of storing them as CI artifacts

Should we try to make it work with just 1 CI provider to start: say GH Actions?

[henryiii]

I've been pretty busy, and just started teaching a class at Princeton this week, too. In a couple of weeks or so the course should settle down to a rhythm (also have PyHEP next week). I'm also just getting started on my (funded!) Scikit-build project. So my time is currently minimal, but should improve later.

• Docker would be slower / more expensive, so if you could avoid it, that would be rather nice. You aren't worried about issues with system packages like we are with manylinux. If a runner didn't have the requirements, we could suggest they run cibuildwheel from the pyodide-env docker container. In fact, we could detect that and set the platform if it's running inside that docker container! :)
• There's a test skip feature already.
• There's precedence for opt-in features - but I don't think we need that, since this will be a new (and only explicit, unless the in-container idea above is used) platform.
• I'd assume you'd normally grab the CI artifacts and upload them to pages as part of some sort of deployment. So not terrible ATM. Once things stabilize, PyPI will probably start accepting them.

I'd work on GHA first, then we can see if we can start adding more - we have a table for what systems support what. It shouldn't be too specific, so should be easy to expand.

One question: as CPython starts getting better support & built binaries, how will Pyodide integrate? One thing users will likely want is to be able to use an "official" release (once it hits tier 2, anyway). Is there some sort of setting we should include, or will it be multiple platforms, or will pyodide and CPython be compatible after 3.11? (I'd guess the main issue would be the Emscripten version matching?)

[hoodmane]

• Docker

Indeed, we don't seem to need it. As long we have a linux OS with appropriate versions of Emscripten, node, and python that should be sufficient. I haven't yet tried to build in very sparse docker containers to determine the exact requirements, but that would probably be a useful exercise. But it works on my host machine and on the default github actions ubuntu. Certainly most of the pyodide-env container is not needed.

• tests

There is good hope that these will eventually be quite simple. Generally it's necessary to skip a few tests that use e.g., subprocess or threads. If it seems worthwhile we could probably even get subprocess working in the test runner (but not threads)

CPython built binaries,

For now all JavaScript needs to be statically linked and our code to load packages, load the runtime, and set up the foreign function interface is all JavaScript. We can look into side loading these eventually, but I think it will be technically difficult. Of course there is also the Emscripten version matching issue as always.

[rth]

Thanks for the update and the pointers! It's was just to get some idea of the next steps, and it's more clear now.

[reynoldsnlp]

Perhaps this issue should be closed since #1454 is a duplicate, but the pull request that @hoodmane is working on (#1456) is connected to that issue.