pyo3
pyo3 copied to clipboard
PyO3
Basic GraalPy Support
With these changes, a GraalPy master build, and using maturin master, I can run the pytests and the examples. I have 3 failures in the pytests still. Please let me know if this is going in the right direction for you.
I think the most important question to clarify here is the status of this integration in the CI. If we are planning on proper support we should include it both for checking PR and for full target checks for the merge queues.
Is this possible and could you sketch an approach here?
I think the most important question to clarify here is the status of this integration in the CI. If we are planning on proper support we should include it both for checking PR and for full target checks for the merge queues. Is this possible and could you sketch an approach here?
There are some changes for this that are only in graalpy master. Our releases go into pyenv, and our next release that will include the relevant changes will be out in September. So one option is to then setup the CI action to add graalpy to the matrix, install it using pyenv and simply run nox -s test-py with it. We can also at any time do nightly builds and we start by adding an action that just uses a well-known nightly build in CI, and we switch to a release build via pyenv once that is out.
install it using pyenv and simply run nox -s test-py with it
Ideally, this would use exactly the same CI jobs we use for CPython and PyPy or is there a technical limitation which makes this unreasonable?
We can also at any time do nightly builds and we start by adding an action that just uses a well-known nightly build in CI, and we switch to a release build via pyenv once that is out.
I think this mainly depends on how urgent this is for you, if you'd like to have this in our main branch before your release, then integrating a nightly build would be a prerequisite. Otherwise, waiting for the next release appears preferable.
install it using pyenv and simply run nox -s test-py with it
Ideally, this would use exactly the same CI jobs we use for CPython and PyPy or is there a technical limitation which makes this unreasonable?
We would first have to get graalpy into github actions.
We can also at any time do nightly builds and we start by adding an action that just uses a well-known nightly build in CI, and we switch to a release build via pyenv once that is out.
I think this mainly depends on how urgent this is for you, if you'd like to have this in our
mainbranch before your release, then integrating a nightly build would be a prerequisite. Otherwise, waiting for the next release appears preferable.
We would of course prefer to have this in before our next release, so that we can start running the tests of actual packages that use PyO3 in our CI and fix any additional issues that occur before the September release.
We are adding support for graalpy to the setup-python action, then running it in the PyO3 CI should need minimal changes
The setup-python action maintainers had a question about GraalPy's licence, I hope they are happy with it soon and will merge, at which point I will rebase this PR and add GraalPy pytest to your CI (https://github.com/actions/setup-python/pull/694)
This now passes all pytests on GraalPy 23.1 with the latest maturin release. Once https://github.com/PyO3/maturin/pull/1773 and https://github.com/actions/setup-python/pull/694 are merged, I can add GraalPy to the CI config.
Great! Do you have a sense of how close the setup-python PR is to merging? I think we're going to push an 0.20 release imminently, so if this is extremely close we could wait for it. (Otherwise this could land in 0.20.1 later.)
Great! Do you have a sense of how close the setup-python PR is to merging? I think we're going to push an 0.20 release imminently, so if this is extremely close we could wait for it. (Otherwise this could land in 0.20.1 later.)
I couldn't say. We've tried to be very quick to do any requested changes, but I assume the github folks are busy and they end up not getting back to us for a week or two each time they ask for something. So don't wait for us.
@timfel I added CI-build-full label, so if you fix the fit issues and push again you should find we do a full CI run including the new graalpy job.
Ok, it's basically working now. The failure on python3.9 is a 500 server error downloading the manylinux image, so I suspect it's transient.
For GraalPy, we need a new release of maturin with this PR: https://github.com/PyO3/maturin/pull/1802. I ran the CI config locally and with an updated maturin it passes that one step that's not running.
Great, I will try to give this a full review in the coming days! I just got back from EuroRust with a ton of threads to follow up before they go cold as well as some time due for my family, so please forgive me if it takes a little longer than usual 🙏
@messense, when might you plan to do the next
maturinrelease?
I'll release a new version after fixing https://github.com/PyO3/maturin/issues/1804 this week.
maturin 1.3.1 is out with the GraalPy wheel filename fix.
looks like the pytests are passing on graalpy with that 🙂
@timfel I think if you rebase you should have a passing CI (nightly is still broken but won't block merge).
Overall it looks to me like this is probably ready to merge, and there's going to be follow-up / cleanup work for a while too. @timfel how much help can we expect from you folks upstream if users are reporting problems here?
Please @-mention me and @msimacek for GraalPy issues. We will continue with the follow up issues and also respond as quickly as we can to any issues here. We are internally running the tests of around 600 Python packages daily on GraalPy, so if people report issues with packages using PyO3, we will also add more tests to our CI.
I think we should also seriously consider working on HPy support in PyO3 in the mid term so that both PyPy and GraalPy implementations can use that, to reduce complexity for everyone.
I agree, we discussed this in the past and I will bring it up again in the next HPy monthly call.
CodSpeed Performance Report
Merging #3247 will not alter performance
Comparing timfel:timfel/graalpy-support-mk2 (c5ae4c7) with main (7d319a9)
Summary
✅ 70 untouched benchmarks
![codspeed-hq[bot] avatar](https://avatars.githubusercontent.com/in/257293?v=4 "Published by a pub.dev verified publisher"){kind=small}
Just looping back to here, I think with 0.21 release the gil-refs removal will not be far enough along to safely support GraalPy; I think that in 0.22 when we fully gate the gil-refs API we can enable GraalPy support as long as that feature is not enabled.
Just looping back to here, I think with 0.21 release the
gil-refsremoval will not be far enough along to safely support GraalPy; I think that in 0.22 when we fully gate thegil-refsAPI we can enable GraalPy support as long as that feature is not enabled.
Thanks. We have (on GraalPy master) now changed our PyObject* representation, and are now allocating and populating PyObject* structures in native memory and handing out the real pointers to that memory to C extensions. So from our side once we do our next release in April we could also simply work with the current codebase of non-opaque structures
With release of GraalPy 24.0 last week we now actually allocate proper memory for PyObject structures. So I made everything else opaque/configured away.
With release of GraalPy 24.0 last week we now actually allocate proper memory for PyObject structures. So I made everything else opaque/configured away.
So assuming conflict resolution, this can be reviewed and possibly merged now and CI should continue to pass against the publicly available versions of GraalPy?
So assuming conflict resolution, this can be reviewed and possibly merged now and CI should continue to pass against the publicly available versions of GraalPy?
Yes 🙂
Do GraalPy wheels get uploaded to PyPI? Does the GraalPy version (presumably 24.0) go in the tag?
Do GraalPy wheels get uploaded to PyPI? Does the GraalPy version (presumably 24.0) go in the tag?
Yes, the graalpy and implemented cpython version go in the tag. pip debug gives wheel tags like graalpy310-graalpy240_310_native-manylinux_2_35_x86_64
I'm not entirely sure how the performance differences (both regressions and improvements) are caused by my changes. Maybe those benchmarks are so micro that there is jitter?
Yep there's a few benchmarks which need to have volatility ignored after I re-enabled them in #3986
I cleared those now 👍
Is the failure in "CI / python3.12-x64 ubuntu-latest rust-nightly" something I could have done? It fails in pyo3-macros-backend, I haven't touched that, but I fear I don't understand enough to rule out some interaction with my changes.
Nightly probably has improved dead code detection. I don't think you need to worry about it here.