Build broken on python 3.13-dev

[aciba90]

rpds-py cannot be built on Python 3.13.0a6+.

Steps to reproduce

pyenv install 3.13-dev 
pyenv virtualenv 3.13-dev rpds-test
pyenv shell rpds-test 

$ pip install rpds-py
Collecting rpds-py
  Downloading rpds_py-0.18.1.tar.gz (25 kB)
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
  Preparing metadata (pyproject.toml) ... done
Building wheels for collected packages: rpds-py
  Building wheel for rpds-py (pyproject.toml) ... error
  error: subprocess-exited-with-error
  
  × Building wheel for rpds-py (pyproject.toml) did not run successfully.
  │ exit code: 1
  ╰─> [52 lines of output]
      Running `maturin pep517 build-wheel -i /home/aciba/.pyenv/versions/3.13-dev/envs/rpds-test/bin/python --compatibility off`
      📦 Including license file "/tmp/pip-install-mpcdnjtn/rpds-py_c225fa28d6e746c5bdcd1b2440b5d7ec/LICENSE"
      🔗 Found pyo3 bindings
      🐍 Found CPython 3.13 at /home/aciba/.pyenv/versions/3.13-dev/envs/rpds-test/bin/python
      📡 Using build options features from pyproject.toml
         Compiling target-lexicon v0.12.9
         Compiling once_cell v1.18.0
         Compiling autocfg v1.1.0
         Compiling proc-macro2 v1.0.66
         Compiling libc v0.2.147
         Compiling unicode-ident v1.0.11
         Compiling parking_lot_core v0.9.8
         Compiling heck v0.4.1
         Compiling scopeguard v1.1.0
         Compiling smallvec v1.11.0
         Compiling cfg-if v1.0.0
         Compiling portable-atomic v1.6.0
         Compiling static_assertions v1.1.0
         Compiling triomphe v0.1.9
         Compiling indoc v2.0.4
         Compiling unindent v0.2.3
         Compiling archery v1.2.0
         Compiling lock_api v0.4.10
         Compiling memoffset v0.9.0
         Compiling rpds v1.1.0
         Compiling pyo3-build-config v0.20.3
         Compiling quote v1.0.31
         Compiling syn v2.0.32
         Compiling parking_lot v0.12.1
         Compiling pyo3-ffi v0.20.3
         Compiling pyo3 v0.20.3
      error: failed to run custom build command for `pyo3-ffi v0.20.3`
      
      Caused by:
        process didn't exit successfully: `/tmp/pip-install-mpcdnjtn/rpds-py_c225fa28d6e746c5bdcd1b2440b5d7ec/target/release/build/pyo3-ffi-70093274310161c8/build-script-build` (exit status: 1)
        --- stdout
        cargo:rerun-if-env-changed=PYO3_CROSS
        cargo:rerun-if-env-changed=PYO3_CROSS_LIB_DIR
        cargo:rerun-if-env-changed=PYO3_CROSS_PYTHON_VERSION
        cargo:rerun-if-env-changed=PYO3_CROSS_PYTHON_IMPLEMENTATION
        cargo:rerun-if-env-changed=PYO3_PRINT_CONFIG
        cargo:rerun-if-env-changed=PYO3_USE_ABI3_FORWARD_COMPATIBILITY
      
        --- stderr
        error: the configured Python interpreter version (3.13) is newer than PyO3's maximum supported version (3.12)
        = help: please check if an updated version of PyO3 is available. Current version: 0.20.3
        = help: set PYO3_USE_ABI3_FORWARD_COMPATIBILITY=1 to suppress this check and build anyway using the stable ABI
      warning: build failed, waiting for other jobs to finish...
      💥 maturin failed
        Caused by: Failed to build a native library through cargo
        Caused by: Cargo build finished with "exit status: 101": `env -u CARGO PYO3_ENVIRONMENT_SIGNATURE="cpython-3.13-64bit" PYO3_PYTHON="/home/aciba/.pyenv/versions/3.13-dev/envs/rpds-test/bin/python" PYTHON_SYS_EXECUTABLE="/home/aciba/.pyenv/versions/3.13-dev/envs/rpds-test/bin/python" "cargo" "rustc" "--features" "pyo3/extension-module" "--message-format" "json-render-diagnostics" "--manifest-path" "/tmp/pip-install-mpcdnjtn/rpds-py_c225fa28d6e746c5bdcd1b2440b5d7ec/Cargo.toml" "--release" "--lib"`
      Error: command ['maturin', 'pep517', 'build-wheel', '-i', '/home/aciba/.pyenv/versions/3.13-dev/envs/rpds-test/bin/python', '--compatibility', 'off'] returned non-zero exit status 1
      [end of output]
  
  note: This error originates from a subprocess, and is likely not a problem with pip.
  ERROR: Failed building wheel for rpds-py
Failed to build rpds-py
ERROR: Could not build wheels for rpds-py, which is required to install pyproject.toml-based projects

[Julian]

I don't personally test on 3.13 yet, though I'm of course happy to -- it's possible that PyO3 0.21 works on it / has a fix for this, but the GIL API changes were definitely not trivial for me, I haven't been able to get something working yet. I pushed my partial attempt to https://github.com/crate-py/rpds/compare/main...pyo3-21 -- PRs are welcome to finish that. Otherwise this will happens when it happens.

[aciba90]

Thanks for the heads-up. I have tried with PyO3 0.21 with the same result. Per https://github.com/PyO3/pyo3/issues/4038, it looks like 3.13 support is a nice to have for PyO3 0.22.

[Julian]

Got it thanks for the pointer. I think the above still will be a blocker then regardless, as presumably they won't backport the 0.22 changes to 0.20 so we'll have to move up.

[FlickerSoul]

Python 3.13 seems to be supported with the new 0.22 release of PyO3 https://github.com/PyO3/pyo3/releases/tag/v0.22.0

Would this be enough to make rpds compatible? Would love to help if possible :)

[edgarrmondragon]

Python 3.13 seems to be supported with the new 0.22 release of PyO3 https://github.com/PyO3/pyo3/releases/tag/v0.22.0

Would this be enough to make rpds compatible? Would love to help if possible :)

@FlickerSoul you might wanna take a look at the failures in https://github.com/crate-py/rpds/pull/76

[Julian]

Yep that PR is definitely what needs help -- I pushed a branch with my own WIP here:

https://github.com/crate-py/rpds/tree/pyo3-21

which is the first part of this (updating to 0.21) -- you may or may not find that helpful to start with.

(n.b. things are undoubtedly slower at the minute on my end. https://www.linkedin.com/feed/update/urn:li:activity:7213574952485089280/ has the quick/not so quick explanation. Just mentioning as I apologize if responses and/or attention takes longer than usual, and/or if you're wondering when or if I'll get to this myself!)

[FlickerSoul]

Thanks for the information! I'll take a look later this week :)

[tacaswell]

Experimentally, this will compile on py3.13 with PYO3_USE_ABI3_FORWARD_COMPATIBILITY = 1 set. Might be worth defaulting to that to make future versions less painful to deal with (but on the other hand, the limited API is not supported with freethreading builds).

[FlickerSoul]

PyO3 removes the Clone trait on PyObject by default. We can re-enable it by adding py-clone feature on PyO3. But the side effect of py-clone is that when clone is called without GIL, the program will panic. The solution in the case without the py-clone flag is to hold GIL explicitly when a PyObject is replicated (or when the reference count is updated).

I made two branches for each case, one without the py-clone feature, and one without.

https://github.com/crate-py/rpds/compare/main...FlickerSoul:rpds:dev

https://github.com/crate-py/rpds/compare/main...FlickerSoul:rpds:clone-feature

Both versions work on the GIL version of CPython and all tests pass. However, they don't work on free-threaded one: Python crashes during PyO3 making the py model. This seems to be a bug in PyO3. I have filed an issue and I guess I can only wait for their responses.

Let me know if I missed anything. :)

Thanks!

[Julian]

Thanks for the work so far!

Two immediate notes:

I notice @edgarrmondragon also sent a draft PR, so it's likely wise to coordinate between the two of you on what's what and which/when they're ready for review.

On the free-threading question: I suspect most Python packages will not explicitly support the free-threading interpreter for the immediate future (as a maintainer I certainly don't plan on it in the short term, only perhaps quite a bit down the line, and certainly not without CI running both interpreters) -- so I'm totally fine sticking to things that work with "Traditional CPython", especially if it gets us more functionality there.

So yeah, I wouldn't intentionally break GIL-less, but if there's something which works on standard CPython we should keep it.

And yeah thanks again for your work so far it will be very helpful I'm sure in getting this over the line!

[FlickerSoul]

Thanks for the quick reply! We will discuss in the PR thread and will let you know when it's ready.

Yes, GitHub doesn't have a GIL-less CPython CI for now (here). I'll keep an eye on it while waiting updates from PyO3 side.

Thanks!

[Julian]

This should be good to go in 0.19. Thanks a ton to @FlickerSoul and @edgarrmondragon.

[logoff]

And 3.13.0 is out, no longer a dev version: https://www.python.org/downloads/release/python-3130/