pyca
Build error on macOS ARM no GIL
Attempting to build on Python 3.14t on macOS on ARM, I get the following failure:
building '_sodium' extension
creating build/temp.macosx-15.1-arm64-cpython-314t/build/temp.macosx-15.1-arm64-cpython-314t
gcc -fno-strict-overflow -Wsign-compare -Wunreachable-code -DNDEBUG -g -O3 -Wall -I/opt/python/include/python3.14t -Ibuild/temp.macosx-15.1-arm64-cpython-314t/include -c build/temp.macosx-15.1-arm64-cpython-314t/_sodium.c -o build/temp.macosx-15.1-arm64-cpython-314t/build/temp.macosx-15.1-arm64-cpython-314t/_sodium.o
In file included from build/temp.macosx-15.1-arm64-cpython-314t/_sodium.c:57:
/opt/python/include/python3.14t/Python.h:51:4: error: "The limited API is not currently supported in the free-threaded build"
51 | # error "The limited API is not currently supported in the free-threaded build"
| ^
In file included from build/temp.macosx-15.1-arm64-cpython-314t/_sodium.c:57:
In file included from /opt/python/include/python3.14t/Python.h:72:
/opt/python/include/python3.14t/object.h:146:5: error: unknown type name 'PyMutex'
146 | PyMutex ob_mutex; // per-object lock
| ^
/opt/python/include/python3.14t/object.h:299:5: error: call to undeclared function '_Py_atomic_store_ssize_relaxed'; ISO C99 and later do not support implicit function declarations [-Wimplicit-function-declaration]
299 | _Py_atomic_store_ssize_relaxed(&ob->ob_size, size);
| ^
In file included from build/temp.macosx-15.1-arm64-cpython-314t/_sodium.c:57:
In file included from /opt/python/include/python3.14t/Python.h:73:
/opt/python/include/python3.14t/refcount.h:97:26: error: call to undeclared function '_Py_atomic_load_uint32_relaxed'; ISO C99 and later do not support implicit function declarations [-Wimplicit-function-declaration]
97 | uint32_t local = _Py_atomic_load_uint32_relaxed(&ob->ob_ref_local);
| ^
/opt/python/include/python3.14t/refcount.h:101:29: error: call to undeclared function '_Py_atomic_load_ssize_relaxed'; ISO C99 and later do not support implicit function declarations [-Wimplicit-function-declaration]
101 | Py_ssize_t shared = _Py_atomic_load_ssize_relaxed(&ob->ob_ref_shared);
| ^
/opt/python/include/python3.14t/refcount.h:115:13: error: call to undeclared function '_Py_atomic_load_uint32_relaxed'; ISO C99 and later do not support implicit function declarations [-Wimplicit-function-declaration]
115 | return (_Py_atomic_load_uint32_relaxed(&op->ob_ref_local) ==
| ^
/opt/python/include/python3.14t/refcount.h:115:63: warning: comparison of integers of different signs: 'int' and 'unsigned int' [-Wsign-compare]
115 | return (_Py_atomic_load_uint32_relaxed(&op->ob_ref_local) ==
| ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ ^
116 | _Py_IMMORTAL_REFCNT_LOCAL);
| ~~~~~~~~~~~~~~~~~~~~~~~~~
/opt/python/include/python3.14t/refcount.h:146:9: error: call to undeclared function '_Py_IsOwnedByCurrentThread'; ISO C99 and later do not support implicit function declarations [-Wimplicit-function-declaration]
146 | if (_Py_IsOwnedByCurrentThread(ob)) {
| ^
/opt/python/include/python3.14t/refcount.h:241:22: error: call to undeclared function '_Py_atomic_load_uint32_relaxed'; ISO C99 and later do not support implicit function declarations [-Wimplicit-function-declaration]
241 | uint32_t local = _Py_atomic_load_uint32_relaxed(&op->ob_ref_local);
| ^
/opt/python/include/python3.14t/refcount.h:248:9: error: call to undeclared function '_Py_IsOwnedByCurrentThread'; ISO C99 and later do not support implicit function declarations [-Wimplicit-function-declaration]
248 | if (_Py_IsOwnedByCurrentThread(op)) {
| ^
/opt/python/include/python3.14t/refcount.h:249:9: error: call to undeclared function '_Py_atomic_store_uint32_relaxed'; ISO C99 and later do not support implicit function declarations [-Wimplicit-function-declaration]
249 | _Py_atomic_store_uint32_relaxed(&op->ob_ref_local, new_local);
| ^
/opt/python/include/python3.14t/refcount.h:252:9: error: call to undeclared function '_Py_atomic_add_ssize'; ISO C99 and later do not support implicit function declarations [-Wimplicit-function-declaration]
252 | _Py_atomic_add_ssize(&op->ob_ref_shared, (1 << _Py_REF_SHARED_SHIFT));
| ^
/opt/python/include/python3.14t/refcount.h:335:22: error: call to undeclared function '_Py_atomic_load_uint32_relaxed'; ISO C99 and later do not support implicit function declarations [-Wimplicit-function-declaration]
335 | uint32_t local = _Py_atomic_load_uint32_relaxed(&op->ob_ref_local);
| ^
/opt/python/include/python3.14t/refcount.h:341:9: error: call to undeclared function '_Py_IsOwnedByCurrentThread'; ISO C99 and later do not support implicit function declarations [-Wimplicit-function-declaration]
341 | if (_Py_IsOwnedByCurrentThread(op)) {
| ^
/opt/python/include/python3.14t/refcount.h:343:9: error: call to undeclared function '_Py_atomic_store_uint32_relaxed'; ISO C99 and later do not support implicit function declarations [-Wimplicit-function-declaration]
343 | _Py_atomic_store_uint32_relaxed(&op->ob_ref_local, local);
| ^
/opt/python/include/python3.14t/refcount.h:345:13: error: call to undeclared function '_Py_MergeZeroLocalRefcount'; ISO C99 and later do not support implicit function declarations [-Wimplicit-function-declaration]
345 | _Py_MergeZeroLocalRefcount(op);
| ^
/opt/python/include/python3.14t/refcount.h:349:9: error: call to undeclared function '_Py_DecRefShared'; ISO C99 and later do not support implicit function declarations [-Wimplicit-function-declaration]
349 | _Py_DecRefShared(op);
| ^
1 warning and 16 errors generated.
error: command '/usr/bin/gcc' failed with exit code 1
[end of output]
note: This error orig
This may or may not be related to #834.
CFFI does not support freethreaded Python at this point, so this is not expected to work, see: https://github.com/python-cffi/cffi/issues/119
If I force the build to use our fork of CFFI:
diff --git a/pyproject.toml b/pyproject.toml
index d6cc581..5c91e55 100644
--- a/pyproject.toml
+++ b/pyproject.toml
@@ -3,7 +3,7 @@
requires = [
"setuptools>=61.0.0,!=74.0.0",
"wheel",
- "cffi>=1.4.1; platform_python_implementation != 'PyPy'",
+ "cffi @ git+https://github.com/quansight-labs/cffi; platform_python_implementation != 'PyPy'",
]
build-backend = "setuptools.build_meta"
After updating from tox to nox: https://github.com/pyca/pynacl/pull/862
I can build and run the tests on the free-threaded build. I don't see any test failures.
It would probably be helpful for our effort to upstream the changes in our fork of cffi if we could come up with multithreaded tests. Mostly to stress-test cffi.
I tried building pynacl with TSAN support but hit a build error in libsodium's make check:
make[3]: Leaving directory '/work/pynacl/build/temp.linux-x86_64-cpython-314t/test/default'
make check-TESTS
make[3]: Entering directory '/work/pynacl/build/temp.linux-x86_64-cpython-314t/test/default'
make[4]: Entering directory '/work/pynacl/build/temp.linux-x86_64-cpython-314t/test/default'
PASS: aead_aegis128l
PASS: aead_aegis256
PASS: aead_aes256gcm
PASS: aead_aes256gcm2
PASS: aead_chacha20poly1305
PASS: aead_chacha20poly13052
PASS: aead_xchacha20poly1305
PASS: auth
PASS: auth2
PASS: auth3
PASS: auth5
PASS: auth6
PASS: auth7
PASS: box
PASS: box2
PASS: box7
PASS: box8
PASS: box_easy
PASS: box_easy2
PASS: box_seal
PASS: box_seed
PASS: chacha20
PASS: codecs
PASS: core1
PASS: core2
PASS: core3
PASS: core4
PASS: core5
PASS: core6
PASS: ed25519_convert
PASS: generichash
PASS: generichash2
PASS: generichash3
PASS: hash
PASS: hash3
PASS: kdf
PASS: keygen
PASS: kx
PASS: metamorphic
PASS: misuse
PASS: onetimeauth
PASS: onetimeauth2
PASS: onetimeauth7
PASS: pwhash_argon2i
PASS: pwhash_argon2id
PASS: randombytes
PASS: scalarmult
PASS: scalarmult2
PASS: scalarmult5
PASS: scalarmult6
PASS: scalarmult7
PASS: scalarmult8
PASS: secretbox
PASS: secretbox2
PASS: secretbox7
PASS: secretbox8
PASS: secretbox_easy
PASS: secretbox_easy2
PASS: secretstream_xchacha20poly1305
PASS: shorthash
PASS: sign
PASS: sodium_core
PASS: sodium_utils
PASS: sodium_version
PASS: stream
PASS: stream2
PASS: stream3
PASS: stream4
PASS: verify1
PASS: sodium_utils2
FAIL: sodium_utils3
PASS: core_ed25519
PASS: core_ristretto255
PASS: kdf_hkdf
PASS: pwhash_scrypt
PASS: pwhash_scrypt_ll
PASS: scalarmult_ed25519
PASS: scalarmult_ristretto255
PASS: siphashx24
PASS: xchacha20
============================================================================
Testsuite summary for libsodium 1.0.20
============================================================================
# TOTAL: 80
# PASS: 79
# SKIP: 0
# XFAIL: 0
# FAIL: 1
# XPASS: 0
# ERROR: 0
============================================================================
See test/default/test-suite.log for debugging.
Some test(s) failed. Please report this to https://github.com/jedisct1/libsodium/issues,
together with the test-suite.log file (gzipped) and your system
information. Thanks.
============================================================================
I don't see an upstream issue mentioning TSAN so maybe no one has hit this before.
That test file has a warning in it if you try to compile with asan and appears to be doing a bunch of signal handler testing.
I opened https://github.com/jedisct1/libsodium/issues/1450. I might hack the libsodium build to ignore that error to see if I can run a test I wrote under TSAN that encrypts a message in chunks in a thread pool as well as the full test suite under pytest-run-parallel. The latter is imperfect because it will skip tests using hypothesis because hypothesis isn't thread-safe.
I ended up finding some races in our fork of CFFI (https://github.com/Quansight-Labs/cffi/issues/2#issuecomment-2840037692). Nothing worrying and nothing that's problematic on the GIL-enabled build. Thanks for sanity checking what I was seeing earlier @reaperhulk.
An update: we now have a version of CFFI that I feel much more confident is thread-safe on the free-threaded build thanks to lots of hard work from @colesbury, @kumaraditya303, and myself.
I'm currently in the process of packaging our fork, which includes renaming the project from CFFI to CFFI-ft and renaming the modules it exports to avoid stepping on upstream's modules or packaging metadata if both happen to be installed simultaneously.
@reaperhulk I'm curious what it would take practically from our side to convince you to add a dependency on CFFI-ft for Python 3.13 and newer.
Unfortunately there's currently no way to declare a dependency only on the free-threaded build in a pyproject.toml file, so at least in the PyPI ecosystem you'd need to depend on CFFI-ft for GIL-enabled Python 3.13 and newer. Hopefully this is all temporary and everyone can switch back to depending on upstream CFFI once we propose changes from our fork. I'm not sure how long that will take - upstream asked us to do real-world testing.
I'm planning to package CFFI-ft for PyPI and conda-forge. Are there other packaging ecosystems you'd like me to do some legwork on?
CFFI 2.0.0 is out, so this is fixed now. Hopefully we'll have a PyNaCl release out shortly to get wheel builds too, but if you have a compiler then the build succeeds now:
± pip install pynacl
Collecting pynacl
Downloading PyNaCl-1.5.0.tar.gz (3.4 MB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 3.4/3.4 MB 11.5 MB/s 0:00:00
Installing build dependencies ... done
Getting requirements to build wheel ... done
Preparing metadata (pyproject.toml) ... done
Requirement already satisfied: cffi>=1.4.1 in /Users/goldbaum/.pyenv/versions/3.14.0rc2t/lib/python3.14t/site-packages (from pynacl) (2.0.0b1)
Requirement already satisfied: pycparser in /Users/goldbaum/.pyenv/versions/3.14.0rc2t/lib/python3.14t/site-packages (from cffi>=1.4.1->pynacl) (2.22)
Building wheels for collected packages: pynacl
Building wheel for pynacl (pyproject.toml) ... done
Created wheel for pynacl: filename=pynacl-1.5.0-cp314-cp314t-macosx_15_0_arm64.whl size=101236 sha256=3db13bcb7af74711b5fe9d15cc8491ca9ae0771ba7476919fe4c036f0fdd6f43
Stored in directory: /Users/goldbaum/Library/Caches/pip/wheels/46/db/8c/218b01e4fbcc47efc6ee41db2c8a110837d9341393e5969c01
Successfully built pynacl
Installing collected packages: pynacl
Successfully installed pynacl-1.5.0