Free-threaded Python support

[adamjstewart]

I would like to test my library with the latest free-threaded Python 3.13t and 3.14t with the GIL disabled. However, netCDF4 does not yet provide wheels for this platform. Even if free-threaded mode is not yet fully supported or taken advantage of, publishing wheels for this platform will allow other upstream authors to test their own code in this new mode.

[dopplershift]

Is there benefit to publishing these when the netcdf c library is not thread safe and calling netcdf functions from multiple threads is likely to lead to segfaults?

[adamjstewart]

@rgommers would know the answer to that better than me.

[rgommers]

Yes, there is quite a bit of benefit, especially for a package like this with a hard-to-obtain build dependency. It can be useful for a couple of different reasons:

1. It makes it easier to start testing netcdf4-python itself. By default, the GIL will be re-enabled for cp314t wheels (conservative, safe), and end users can opt in to free-threading by setting the PYTHON_GIL=0 env var.
2. It makes other packages that depend on this one much easier to install. I suspect most users who want to install netcdf4-python under a free-threaded interpreter aren't going to load NetCDF4 files in parallel, they just want regular single-threaded file loading to work so they can use free-threading with their own code without having to build netcdf4-python from source.

@adamjstewart falls in category (2) I suspect: he wants to work on torchgeo's free-threading support, so only needs netcdf4-python to install easily.

[dopplershift]

Seems reasonable. Just wanted to make sure expectations were in line with reality.

[ocefpaf]

Aren't we providing a footgun in 2? Sure, those who know will use it in a single thread, but the number of issues due to the fact that netcdf-c is not thread safe is already high. My guess is that those would only increase if it is provided a a wheel for the free-threaded Python.

[rgommers]

Aren't we providing a footgun in 2?

No, not really, it only makes install easier. Note that if you just add a wheel without making any changes to make the extension modules thread-safe (e.g., add locks around netcdf-c access), then the GIL will be re-enabled on import so that would be no more and no less unsafe than a default (with-GIL) build when used from multiple threads (of course, no faster either).

Only after an audit and testing for thread-safety are done do you then want to update the extension module initialization to keep the GIL disabled at runtime.

---

I think this package is in a very similar position to h5py, which also has to deal with the underlying C library (libhdf5) not being thread-safe when used with Python-level threading. To do so, it has internal locking. Does netcdf-python have internal locks already?

[jswhit]

no netcdf4-python does not have internal locks

[rgommers]

Then it might make sense indeed to consider uploading cp314t wheels (cp313t is experimental and not useful) while leaving the GIL enabled. It's not super difficult for downstream packages to build from source, but figuring out the sudo apt install libhdf5-dev libnetcdf-dev equivalent on all platforms is enough of a pain that it'd be helpful with downstream development work.