pola-rs
Build fails with free-threaded Python
Checks
- [x] I have checked that this issue has not already been reported.
- [x] I have confirmed this bug exists on the latest version of Polars.
Issue description
When building polars against free-threaded Python 3.13.2 on Linux (using the nightly version of Rust), the build fails with the error:
Compiling numpy v0.23.0
error[E0432]: unresolved import `pyo3::sync::GILProtected`
--> /.../.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/numpy-0.23.0/src/datetime.rs:63:12
|
63 | use pyo3::{sync::GILProtected, Bound, Py, Python};
| ^^^^^^^^^^^^^^^^^^ no `GILProtected` in `sync`
|
note: found an item that was configured out
--> /.../.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/pyo3-0.23.5/src/sync.rs:47:12
|
47 | pub struct GILProtected<T> {
| ^^^^^^^^^^^^
note: the item is gated here
--> /.../.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/pyo3-0.23.5/src/sync.rs:46:1
|
46 | #[cfg(not(Py_GIL_DISABLED))]
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
error[E0432]: unresolved import `pyo3::sync::GILProtected`
--> /.../.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/numpy-0.23.0/src/strings.rs:15:5
|
15 | sync::GILProtected,
| ^^^^^^^^^^^^^^^^^^ no `GILProtected` in `sync`
|
= note: unresolved item `crate::datetime::units::GILProtected` exists but is inaccessible
note: found an item that was configured out
--> /.../.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/pyo3-0.23.5/src/sync.rs:47:12
|
47 | pub struct GILProtected<T> {
| ^^^^^^^^^^^^
note: the item is gated here
--> /.../.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/pyo3-0.23.5/src/sync.rs:46:1
|
46 | #[cfg(not(Py_GIL_DISABLED))]
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
error[E0282]: type annotations needed for `&mut _`
--> /.../.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/numpy-0.23.0/src/datetime.rs:229:13
|
229 | let dtype = match dtypes.get_or_insert_with(Default::default).entry(unit) {
| ^^^^^
...
247 | dtype.bind(py).to_owned()
| ---- type must be known at this point
|
help: consider giving `dtype` an explicit type, where the type for type parameter `V` is specified
|
229 | let dtype: &mut V = match dtypes.get_or_insert_with(Default::default).entry(unit) {
| ++++++++
error[E0282]: type annotations needed for `&mut _`
--> /.../.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/numpy-0.23.0/src/strings.rs:185:13
|
185 | let dtype = match dtypes.get_or_insert_with(Default::default).entry(size) {
| ^^^^^
...
198 | dtype.bind(py).to_owned()
| ---- type must be known at this point
|
help: consider giving `dtype` an explicit type, where the type for type parameter `V` is specified
|
185 | let dtype: &mut V = match dtypes.get_or_insert_with(Default::default).entry(size) {
| ++++++++
Some errors have detailed explanations: E0282, E0432.
For more information about an error, try `rustc --explain E0282`.
error: could not compile `numpy` (lib) due to 4 previous errors
warning: build failed, waiting for other jobs to finish...
💥 maturin failed
Caused by: Failed to build a native library through cargo
Caused by: Cargo build finished with "exit status: 101": `env -u CARGO PYO3_ENVIRONMENT_SIGNATURE="cpython-3.13-64bit" PYO3_PYTHON="/.../.cache/uv/builds-v0/.tmpV8LOZL/bin/python"
PYTHON_SYS_EXECUTABLE="/.../.cache/uv/builds-v0/.tmpV8LOZL/bin/python" "cargo" "rustc" "--message-format" "json-render-diagnostics" "--manifest-path" "/.../.cache/uv/sdists-v8/pypi/polars/1.25.2/ItmSIH_8XlFqKYzSiXKdr/src/py-polars/Cargo.toml"
"--release" "--lib"`
Error: command ['maturin', 'pep517', 'build-wheel', '-i', '/.../.cache/uv/builds-v0/.tmpV8LOZL/bin/python', '--compatibility', 'off'] returned non-zero exit status 1
This error has already been solved upstream by https://github.com/PyO3/rust-numpy/pull/471 and released in rust-numpy 0.24.
To resolve this, I believe that at least polars needs to upgrade the Rust dependency numpy to 0.24 (and likely pyo3 to 0.24 too). Other fixes may be necessary too.
This upgrade work may be related to the PRs #20111 and #20683 by @bschoenmaeckers .
It is currently not possible to upgrade to the latest PyO3 release because of a upstream bug. I will pick this up when ready. Free-threading probably requires more internal changes to prevent deadlocks.
I did some testing in #21914 and the main blockers in our dependency tree are:
- [ ] cffi
- ongoing work at https://github.com/quansight-labs/cffi
- [ ] cython
- will be released under v3.1.0
- [ ] gevent which depends on cython
- [ ] greenlet
- [ ] cryptography which depends on cffi
After this we should be able to do some testing in the CI to make sure polars is thread-safe.
Note that greenlet is only a dependency of gevent and it's not used directly. Also given how they're maintained (not abandoned, but mostly catching up with upstream python) and how they work (by exploiting python internals and swapping stacks) I doubt that we will see them support free-threaded in foreseeable future.
Also they're used only for collect_async (which are marked experimental) as an alternative mode to asyncio, that I made, because I needed that for a legacy project, that since migrated to not using gevent 😃 My point is while they're default libraries for green threading - green threading in general is not a popular choice in python nowadays and my feeling that you can count users that use polars+gevent+collect_async on one hand (if there are any). So, removing gevent support in collect_async altogether should be fine?
But seems like they shouldn't block free-threaded builds, because its an optional dependency and you couldn't build/use them in free-threaded anyway. But they're have to be excluded from requirements/tests in free-threaded in CI.
ref https://github.com/python-greenlet/greenlet/issues/423 https://github.com/gevent/gevent/issues/2087
#22015 is merged, so it is now possible to install the polars source-distribution on free-threaded python. The module is not marked as free-threaded supported but you can force disable the gil by setting the env var to PYTHON_GIL=0.
Hi, I'm a bit of a noob when it comes to the intricacies of low-level things like this, any idea of when the main polars branch will be able to intuitively support free threading in Python?
Also, if anyone would be able to help (I would be very grateful :D) had a quick question regarding using multi-processing with Polars (I'm using Polars for the first time vs Pandas) as I have several different processes from the same Python app running on my one server (with ~14 cores):
I've read that polars will attempt to use ALL (free/unused?) cores for speed, but obviously this will present problems if all of my 1000 forked/spawned Python processes try to each use as many cores as they can because of polars, so I was hoping the free-threading update would prevent that (allowing Python to finally unlock multi-threading computations). Is my understanding of how multi-processing and Polars work correct or am I misunderstanding something?
How would the free-threading update play into this?
I see there's some worry about stability over at https://github.com/pola-rs/polars/pull/22466#issuecomment-2835533010.
Out of curiosity @orlp what would it take for you to feel comfortable calling polars thread-safe and supported under the free-threaded build? I helped add support for the free-threaded build in PyO3 and concerns from Polars could certainly influence further upstream development.
(cc @colesbury @davidhewitt)
@ngoldbaum As a bare minimum it should pass our CI test suite. There may be an exception or two for tests which fail due to unsupported dependencies for specific tests.
@ngoldbaum I've started working on CI coverage for the free-threaded build at #21914. But it stalled on lack of support from downstream dependencies. I will update into updating that MR in the near future. But help is always welcome!
@bschoenmaeckers Thank you for the work! I have a question on stackoverflow, and I am wondering if building Polars from source against Python3.14t is the solution?
@bschoenmaeckers Thank you for the work! I have a question on stackoverflow, and I am wondering if building Polars from source against Python3.14t is the solution?
Free-threaded python is not necessary for polars to do its magic. It already uses all available threads to do the internal processes. It is not tested so you might run into deadlocks, so that may be one pitfall. That said, if you do not experience any deadlocks free-threaded python may speed up other python code running in other threads.
FYI; the installation takes a long time because we do not provide prebuilt wheels for free-threaded python so it has to compile a custom build locally.