Add cp313t tests, mark extension modules as compatible, and ship free-threaded wheels

[ngoldbaum]

Now that https://github.com/huggingface/tokenizers/issues/1767 is closed, tokenizers builds under the free-threaded build.

aiohttp is an (indirect) dev dependency, and they're still working on support, so that might be a complication.

The extension module also still needs to be marked as compatible with the free-threaded build:

<frozen importlib._bootstrap>:488
  <frozen importlib._bootstrap>:488: RuntimeWarning: The global interpreter lock (GIL) has been enabled to load module 'tokenizers.tokenizers', which has not declared that it can run safely without the GIL. To override this behavior and keep the GIL disabled (at your own risk), run with PYTHON_GIL=0 or -Xgil=0.

If I do PYTHON_GIL=0 make test, the tests all pass. That said, I don't see any explicitly multithreaded tests, so it might be a good idea to extend the multiprocessing tests to also use thread pools, to the extent that's feasible. It might also be a good idea to try pytest-run-parallel to shake out any use of global state in the tests or implementation.

[ngoldbaum]

I looked more at free-threaded support today. I looked a little at making the pyclasses in the python bindings frozen to opt out of runtime borrow-checking, but that is currently blocked on the extensive use of __setstate__ internally. Since __setstate__ takes &mut self, it's incompatible with a frozen pyclass. I think tokenizers would need to migrate to using e.g. __reduce__ or __getnewargs_eg__ to avoid that.

There are a few thread-unsafe tests with pytest-run-parallel due to the way the multiprocessing tests manipulate os.environ, some use of the filesystem that assumes no other instances of a test are running, and a couple cases where a fixture that gets shared between threads and mutated. If I fix these issues or mark tests that have to use global state as thread-unsafe, then pytest-run-parallel passes.

I'm planning to open a PR shortly that adds 3.13t CI and runs the tests under pytest-run-parallel with PYTHON_GIL=0, along with a few minor fixes. We can then declare free-threaded support in the pymodules and add free-threaded wheel builds as a next step.

[ngoldbaum]

I'm planning to open a PR shortly

see #1809

[ngoldbaum]

I tried testing on 3.14.0t RC1 but it looks like that's blocked on hf-xet updating to a newer PyO3:

        --- stderr
        error: The configured Python interpreter version (3.14) is newer than PyO3's maximum supported version (3.13)
        = help: please check if an updated version of PyO3 is available. Current version: 0.23.5
        = help: The free-threaded build of CPython does not support the limited API so this check cannot be suppressed.
      warning: build failed, waiting for other jobs to finish...
      💥 maturin failed
        Caused by: Failed to build a native library through cargo
        Caused by: Cargo build finished with "exit status: 101": `env -u CARGO MACOSX_DEPLOYMENT_TARGET="11.0" PYO3_ENVIRONMENT_SIGNATURE="cpython-3.14-64bit" PYO3_PYTHON="/Users/goldbaum/Documents/tokenizers/bindings/python/.env/bin/python" PYTHON_SYS_EXECUTABLE="/Users/goldbaum/Documents/tokenizers/bindings/python/.env/bin/python" "cargo" "rustc" "--features" "pyo3/extension-module" "--message-format" "json-render-diagnostics" "--manifest-path" "/private/var/folders/nk/yds4mlh97kg9qdq745g715rw0000gn/T/pip-install-7z3qz8fc/hf-xet_aac961a8a1f14a80876e7506512a8fb4/hf_xet/Cargo.toml" "--release" "--lib" "--" "-C" "link-arg=-undefined" "-C" "link-arg=dynamic_lookup" "-C" "link-args=-Wl,-install_name,@rpath/hf_xet.abi3.so"`
      Error: command ['maturin', 'pep517', 'build-wheel', '-i', '/Users/goldbaum/Documents/tokenizers/bindings/python/.env/bin/python', '--compatibility', 'off'] returned non-zero exit status 1

[ngoldbaum]

see https://github.com/huggingface/tokenizers/pull/1809

We ended up closing this. I opened https://github.com/huggingface/tokenizers/pull/1864 with a more minimal approach and I'm still waiting on review.

hf-xet is making some movement to support the free-threaded build and Python 3.14: https://github.com/huggingface/xet-core/pull/524

[davidhewitt]

I am making progress towards this, probably targeting 3.14t instead of 3.13t (as 3.14 is the first supported release of free-threading in CPython). Started with #1901