Update pyo3 to 0.25.0 and add support for Python 3.14.0b2

[cdce8p]

I've pulled in the latest changes from PyO3 0.25.0 and applied the remaining patch from the current master branch again, https://github.com/ijl/orjson/commit/27d7e27fe16e68fa0ddb43265db5282e038d2a98. As for the other two changes:

• https://github.com/ijl/orjson/commit/d71e2ad598687dc35e9245478276b068ffdb5dd4 with PyTupleObject.ob_hash is part of 0.25.0.
• https://github.com/ijl/orjson/commit/5ae468a084a9b44eda25fe4314b92cd679315c11 with PyASCIIObject.state is a bit more complicated. In https://github.com/PyO3/pyo3/pull/5133 it was decided to no longer expose PyASCIIObjectState. To work around it, I replaced the main uses with PyUnicode_DATA, PyUnicode_GET_LENGTH, and PyUnicode_KIND as well as using unicode_to_str_via_ffi unconditionally on Python 3.14.

Additional changes were necessary for _Py_IsImmortal which was made private in https://github.com/PyO3/pyo3/pull/4811 and _PyDict_SetItem_KnownHash_LockHeld which isn't part of PyO3.

Fixes #569

[cdce8p]

Tests pass with 3.14.0b2 as well.

[musicinmybrain]

This is working in Fedora to build python-orjson for Python 3.14 – thank you! – except that we are still affected by https://github.com/PyO3/pyo3/issues/5175 on i686. That will be fixed upstream in pyo3-ffi by https://github.com/PyO3/pyo3/pull/5180. When you have a chance, could you please update this PR with that fix as well?

[cdce8p]

That will be fixed upstream in pyo3-ffi by PyO3/pyo3#5180. When you have a chance, could you please update this PR with that fix as well?

Added the patch here as well though ideally this would be merged upstream and pyo3 would release another version "soonish".

[musicinmybrain]

Added the patch here as well though ideally this would be merged upstream and pyo3 would release another version "soonish".

Thanks! According to https://github.com/PyO3/pyo3/issues/5175#issuecomment-2939918138, a prompt new release is planned.

[clin1234]

Build error with your branch via pip install:

(venv) cc@pretty:~$ pip install git+https://github.com/cdce8p/[email protected]
Collecting git+https://github.com/cdce8p/[email protected]
  Cloning https://github.com/cdce8p/orjson (to revision python-3.14) to /tmp/pip-req-build-ztb6monb
  Running command git clone --filter=blob:none --quiet https://github.com/cdce8p/orjson /tmp/pip-req-build-ztb6monb
  Running command git checkout -b python-3.14 --track origin/python-3.14
  Switched to a new branch 'python-3.14'
  branch 'python-3.14' set up to track 'origin/python-3.14'.
  Resolved https://github.com/cdce8p/orjson to commit ea5a7d83f4ff5262f131d37dd1d490a634ce2626
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
  Preparing metadata (pyproject.toml) ... done
Building wheels for collected packages: orjson
  Building wheel for orjson (pyproject.toml) ... error
  error: subprocess-exited-with-error
  
  × Building wheel for orjson (pyproject.toml) did not run successfully.
  │ exit code: 1
  ╰─> [67 lines of output]
      Running `maturin pep517 build-wheel -i /home/cc/venv/bin/python3.14t --compatibility off`
      📦 Including license file "/tmp/pip-req-build-ztb6monb/LICENSE-APACHE"
      📦 Including license file "/tmp/pip-req-build-ztb6monb/LICENSE-MIT"
      🍹 Building a mixed python/rust project
      🔗 Found pyo3-ffi bindings
      🐍 Found CPython 3.14t at /home/cc/venv/bin/python3.14t
         Compiling target-lexicon v0.13.2
         Compiling proc-macro2 v1.0.95
         Compiling unicode-ident v1.0.18
         Compiling once_cell v1.21.3
         Compiling serde v1.0.219
         Compiling rustversion v1.0.20
         Compiling libc v0.2.172
         Compiling cfg-if v1.0.0
         Compiling shlex v1.3.0
         Compiling version_check v0.9.5
         Compiling serde_json v1.0.140
         Compiling ryu v1.0.20
         Compiling itoa v1.0.15
         Compiling memchr v2.7.4
         Compiling static_assertions v1.1.0
         Compiling uuid v1.16.0
         Compiling xxhash-rust v0.8.15
         Compiling itoap v1.0.1
         Compiling jiff v0.2.10
         Compiling simdutf8 v0.1.5
         Compiling smallvec v1.15.0
         Compiling half v2.6.0
         Compiling encoding_rs v0.8.35
         Compiling associative-cache v2.0.0
         Compiling bytecount v0.6.8
         Compiling cc v1.2.20
         Compiling castaway v0.2.3
         Compiling pyo3-build-config v0.25.0 (/tmp/pip-req-build-ztb6monb/include/pyo3/pyo3-build-config)
         Compiling quote v1.0.40
         Compiling syn v2.0.101
         Compiling pyo3-ffi v0.25.0 (/tmp/pip-req-build-ztb6monb/include/pyo3/pyo3-ffi)
         Compiling orjson v3.10.18 (/tmp/pip-req-build-ztb6monb)
         Compiling serde_derive v1.0.219
         Compiling compact_str v0.9.0
      error[E0599]: no function or associated item named `new` found for struct `PyMutex` in the current scope
        --> src/ffi/fragment.rs:75:46
         |
      75 |                 ob_mutex: pyo3_ffi::PyMutex::new(),
         |                                              ^^^ function or associated item not found in `PyMutex`
      
      error[E0560]: struct `pyo3_ffi::PyObject` has no field named `_padding`
         --> src/ffi/fragment.rs:124:21
          |
      124 |                     _padding: 0,
          |                     ^^^^^^^^ `pyo3_ffi::PyObject` does not have this field
          |
          = note: available fields are: `ob_flags`
      
      error[E0599]: no function or associated item named `new` found for struct `PyMutex` in the current scope
         --> src/ffi/fragment.rs:126:50
          |
      126 |                     ob_mutex: pyo3_ffi::PyMutex::new(),
          |                                                  ^^^ function or associated item not found in `PyMutex`
      
      Some errors have detailed explanations: E0560, E0599.
      For more information about an error, try `rustc --explain E0560`.
      error: could not compile `orjson` (lib) due to 3 previous errors
      💥 maturin failed
        Caused by: Failed to build a native library through cargo
        Caused by: Cargo build finished with "exit status: 101": `env -u CARGO PYO3_ENVIRONMENT_SIGNATURE="cpython-3.14-64bit" PYO3_PYTHON="/home/cc/venv/bin/python3.14t" PYTHON_SYS_EXECUTABLE="/home/cc/venv/bin/python3.14t" "cargo" "rustc" "--message-format" "json-render-diagnostics" "--manifest-path" "/tmp/pip-req-build-ztb6monb/Cargo.toml" "--release" "--lib"`
      Error: command ['maturin', 'pep517', 'build-wheel', '-i', '/home/cc/venv/bin/python3.14t', '--compatibility', 'off'] returned non-zero exit status 1
      [end of output]
  
  note: This error originates from a subprocess, and is likely not a problem with pip.
  ERROR: Failed building wheel for orjson
Failed to build orjson
ERROR: Failed to build installable wheels for some pyproject.toml based projects (orjson)

[cdce8p]

@clin1234 Which platform are you using? That might help reproduce the error. It works fine on my local machine with cp314-cp314-macosx_11_0_arm64.

[clin1234]

@clin1234 Which platform are you using? That might help reproduce the error. It works fine on my local machine with cp314-cp314-macosx_11_0_arm64.

CPython 3.14t Linux x86_64

[cdce8p]

@clin1234 Pushed a few more commits to fix 3.14t. Would you mind checking it again?

[clin1234]

@clin1234 Pushed a few more commits to fix 3.14t. Would you mind checking it again?

Builds successfully on my end.

[cdce8p]

For PyMutex, would it not be simpler to use std::mem::zeroed() for initializing ob_mutex?

Tbh I just wanted to reuse what's already there. PyMutex::new() is patched in currently, so I just copied that. https://github.com/ijl/orjson/blob/4b29943c2df5035f7ec573c1a0052c08d293c119/include/pyo3/pyo3-ffi/src/cpython/lock.rs#L11-L18

[clin1234]

For PyMutex, would it not be simpler to use std::mem::zeroed() for initializing ob_mutex?

Tbh I just wanted to reuse what's already there. PyMutex::new() is patched in currently, so I just copied that.

https://github.com/ijl/orjson/blob/4b29943c2df5035f7ec573c1a0052c08d293c119/include/pyo3/pyo3-ffi/src/cpython/lock.rs#L11-L18

I suggested it mainly because of https://github.com/python/cpython/blob/main/Include/cpython/lock.h#L22-L31 and https://docs.python.org/3/c-api/init.html#c.PyMutex

[cdce8p]

@ijl Saw your commits yesterday. Just wanted to highlight this PR in case you'd like to reuse parts of it.

[grantlouisherman]

Hey all, I am waiting on this diff to land. Do we know when this will land?

[sh-rp]

same here!

[cdce8p]

Not stale

[cdce8p]

Not stale

[ijl]

3.11.0 is compatible with 3.14 beta 4. PyStr in 75331b7cded71cb7be7ba53f43d4d7bfe3e50652 and ffi::compat 91f946d8f0d5ed4ddc9209f5e99105e0b34ceb57 are different approach.

[cdce8p]

3.11.0 is compatible with 3.14 beta 4.

Awesome, thanks a lot for working on it @ijl! AFAICT this release is looking fine. Testing so far looks good. One note, @musicinmybrain mentioned an issue with i686 earlier which was fixed in Pyo3 0.25.1. I believe the release is still based on 0.25.0.

[musicinmybrain]

One note, @musicinmybrain mentioned an issue with i686 earlier which was fixed in Pyo3 0.25.1. I believe the release is still based on 0.25.0.

The version bound is 0.25,

https://github.com/ijl/orjson/blob/3b03a4ab494eddaf7e8a7b9f140e66105656fd68/Cargo.toml#L55

which would allow 0.25.0, but Cargo.lock has 0.25.1

https://github.com/ijl/orjson/blob/3b03a4ab494eddaf7e8a7b9f140e66105656fd68/Cargo.lock#L175-L177

and the vendored crate in include/cargo/ in the PyPI sdist is version 0.25.1, so I think nobody should encounter problems in practice unless they go out of their way to build with 0.25.0 in particular.