PyO3
Add js_string field when in Pyodide >0.25
I'm hoping to add an extra js_string field to Py_Unicode in Pyodide to allow us to cache Python to js string conversions. This updates pyo3-ffi to know about this field. See https://github.com/pyodide/pyodide/pull/4130
Hmm, how do you plan for this cfg feature to be applied? Does pyodide have its own abi tag? (I can't remember, does pyodide download wasm builds from pypi?)
Hmm, how do you plan for this
cfgfeature to be applied? Doespyodidehave its own abi tag? (I can't remember, does pyodide download wasm builds from pypi?)
I suspect via direct manipulation of the compiler flags, e.g. setting RUSTFLAGS, and adding --cfg pyodide_0_25?
But whatever the plan, I think this would need independent documentation here in pyo3-ffi as to how this is supposed to be integrated or ideally what the general story for building projects using PyO3 and Pyodide is.
I wonder if we should make a separate pyodide_jsstring_cache cfg or something like that that's more specifically aimed at this specific feature.
The plan is for Pyodide wheels to get an abi platform tag like pyodide_202308_0 or something. We also want to ban abi3 wheels. But as of yet Pyodide wheels cannot be uploaded to pypi.
what the general story for building projects using PyO3 and Pyodide is.
Currently to build projects with Pyodide you need to set some funny RUSTFLAGS. I did some work before on reducing this by adjusting the rust emscripten target but it's a lot of work and sticking extra stuff in RUSTFLAGS is easy. Currently maturin and Pyodide build both know what needs to be put there. It would be nice to document this, probably we should put it in Pyodide's docs and xref that from pyo3?
I think version specific tags are slightly better because if the feature is ever retired (or we drop support for too old pyodide) then we can adjust. Do you emit just --cfg pyodide too? Then if we ever drop PyOdide 0.25 support we could replace this with #[cfg(pyodide)].
It would be nice to document this, probably we should put it in Pyodide's docs and xref that from pyo3?
I think that would be a welcome idea!
More generally, we should probably consider what Pyodide version support looks like if we're going to be maintaining pyodide-specific treatment in PyO3. Do Pyodide versions have any kind of rough mapping to CPython releases? I guess in 0.X releases you don't support older releases, so I wonder whether we would need to either?
Do Pyodide versions have any kind of rough mapping to CPython releases?
Yeah, we try to update in the spring releases, but the history isn't completely regular:
0.16 -- 3.8 Released December 25, 2020
0.18 -- 3.9 Released August 3, 2021
0.20 -- 3.10 Released April 9, 2022
0.23 -- 3.11 Released March 30, 2023
Our goal is to keep updates between mid march and mid april. But it would be useful to keep support in pyo3 slightly longer since sometimes people want to use old versions of Pyodide -- the Pyodide updates are offering fewer benefits other than version bumps these days.
Thanks. @hoodmane would you be willing to place that mapping in documentation somewhere? That way, we can easily track what minimum pyodide version to support based on the equivalent CPython support we offer. (And I guess we test minimum & latest pyodides in CI, if it makes sense to do so.)
Otherwise, I guess this just needs a CHANGELOG entry and then we can consider it for merging whenever your upstream PR is ready to ship.
Maybe we should call this something more specific like --cfg pyodide-js-string?
No strong opinion. If we do --cfg pyodide-js-string that then every tool which is used to build needs to know the mapping of Pyodide versions to when this patch applies. Maybe a way to avoid that would be if Pyodide can supply the RUSTFLAGS which it wants to be built with, and build tools can query that?
Pyodide can supply the RUSTFLAGS which it wants to be built with, and build tools can query that?
This sounds super reasonable. I think @ryanking13 had a similar idea.
This sounds super reasonable. I think @ryanking13 had a similar idea.
Maybe we can add an entry for RUSTFLAGS in pyodide config command? e.g.
$ pyodide config rustflags
-C link-arg=-sWASM_BIGINT -C ...
If users are expected to have the pyodide CLI on their command line when building for pyodide, then that sounds good to me too. I think we'd have to figure out how to make maturin and setuptools-rust be aware they're building for pyodide and invoke the CLI to get flags, but that's just a UX question.
I guess that can work. I guess eventually if they're going to be building wheels, they must be aware of the abi / platform?
cc @messense for opinions on how maturin might want to detect pyodide builds.