Add a TSAN instrumented free-threaded CI run

Open ngoldbaum opened this issue 8 months ago • 1 comments

See https://github.com/PyO3/pyo3/issues/4904

Now that the main branch supports 3.14, this should be more useful for upstream if we find bugs.

Apr 27 '25 12:04 ngoldbaum

I was able to get this to work on my Mac now that homebrew has updated to LLVM 20.

I created a free-threaded build of CPython using pyenv with the following command:

CONFIGURE_OPTS="--with-thread-sanitizer" CC=/opt/homebrew/opt/llvm/bin/clang-20 CXX=/opt/homebrew/opt/llvm/bin/clang++-20 pyenv install -v 3.14.0a7t -k -g

I then set up a nightly rust toolchain with the rust-src component and ran the tests with the following command:

RUSTFLAGS="-Zexternal-clangrt" cargo test --no-fail-fast --test test_gc --target aarch64-apple-darwin

I find that about 1/5 of the time, I see races in pyclass internals:

WARNING: ThreadSanitizer: data race (pid=82643)
  Atomic write of size 8 at 0x000116040020 by thread T12 (mutexes: write M0):
    #0 _Py_atomic_add_ssize pyatomic_gcc.h:63 (libpython3.14td.dylib:arm64+0x2c128c)
    #1 Py_INCREF refcount.h:291 (libpython3.14td.dylib:arm64+0x2c114c)
    #2 _Py_NewRef refcount.h:542 (libpython3.14td.dylib:arm64+0x2be8f8)
    #3 PyTuple_Pack tupleobject.c:180 (libpython3.14td.dylib:arm64+0x2be848)
    #4 get_bases_tuple typeobject.c:4771 (libpython3.14td.dylib:arm64+0x2c9f0c)
    #5 PyType_FromMetaclass typeobject.c:5024 (libpython3.14td.dylib:arm64+0x2c8e80)
    #6 PyType_FromSpec typeobject.c:5326 (libpython3.14td.dylib:arm64+0x2cbb68)
    #7 pyo3::pyclass::create_type_object::PyTypeBuilder::build::hf1a96bf6ff0a27e0 <null> (test_gc-8186109d9721178d:arm64+0x10007e660)

  Previous write of size 8 at 0x000116040020 by thread T11 (mutexes: write M1):
    #0 _PyObject_SetDeferredRefcount object.c:2586 (libpython3.14td.dylib:arm64+0x264fb4)
    #1 _PyObject_AssignUniqueId uniqueid.c:96 (libpython3.14td.dylib:arm64+0x66c8cc)
    #2 PyType_FromMetaclass typeobject.c:5230 (libpython3.14td.dylib:arm64+0x2c99e0)
    #3 PyType_FromSpec typeobject.c:5326 (libpython3.14td.dylib:arm64+0x2cbb68)
    #4 pyo3::pyclass::create_type_object::PyTypeBuilder::build::hf1a96bf6ff0a27e0 <null> (test_gc-8186109d9721178d:arm64+0x10007e660)

  Mutex M0 (0x000109c017c0) created at:
    #0 pthread_mutex_init <null> (libclang_rt.tsan_osx_dynamic.dylib:arm64+0x35db8)
    #1 std::sys::pal::unix::sync::mutex::Mutex::init::h5ed2ad5ecd6004b9 <null> (test_gc-8186109d9721178d:arm64+0x1000b8ec4)

  Mutex M1 (0x000109c01880) created at:
    #0 pthread_mutex_init <null> (libclang_rt.tsan_osx_dynamic.dylib:arm64+0x35db8)
    #1 std::sys::pal::unix::sync::mutex::Mutex::init::h5ed2ad5ecd6004b9 <null> (test_gc-8186109d9721178d:arm64+0x1000b8ec4)

  Thread T12 (tid=53509925, running) created by main thread at:
    #0 pthread_create <null> (libclang_rt.tsan_osx_dynamic.dylib:arm64+0x34418)
    #1 std::sys::pal::unix::thread::Thread::new::h95c43de68cf4a977 <null> (test_gc-8186109d9721178d:arm64+0x1000b90f0)

  Thread T11 (tid=53509924, running) created by main thread at:
    #0 pthread_create <null> (libclang_rt.tsan_osx_dynamic.dylib:arm64+0x34418)
    #1 std::sys::pal::unix::thread::Thread::new::h95c43de68cf4a977 <null> (test_gc-8186109d9721178d:arm64+0x1000b90f0)

SUMMARY: ThreadSanitizer: data race pyatomic_gcc.h:63 in _Py_atomic_add_ssize

WARNING: ThreadSanitizer: data race (pid=82689)
  Read of size 8 at 0x000114040040 by thread T3 (mutexes: write M0):
    #0 _PyObject_VAR_SIZE objimpl.h:24 (libpython3.14td.dylib:arm64+0x2c7644)
    #1 _PyType_AllocNoTrack typeobject.c:2293 (libpython3.14td.dylib:arm64+0x2c7414)
    #2 PyType_GenericAlloc typeobject.c:2331 (libpython3.14td.dylib:arm64+0x2c7360)
    #3 _$LT$pyo3..impl_..pyclass_init..PyNativeTypeInitializer$LT$T$GT$$u20$as$u20$pyo3..impl_..pyclass_init..PyObjectInit$LT$T$GT$$GT$::into_new_object::inner::h607105330924cb56 <null> (test_gc-8186109d9721178d:arm64+0x10008dbb0)

  Previous write of size 8 at 0x000114040040 by thread T4 (mutexes: write M1):
    #0 PyType_FromMetaclass typeobject.c:5162 (libpython3.14td.dylib:arm64+0x2c9558)
    #1 PyType_FromSpec typeobject.c:5326 (libpython3.14td.dylib:arm64+0x2cbb68)
    #2 pyo3::pyclass::create_type_object::PyTypeBuilder::build::hf1a96bf6ff0a27e0 <null> (test_gc-8186109d9721178d:arm64+0x10007e660)

  Mutex M0 (0x000105601dc0) created at:
    #0 pthread_mutex_init <null> (libclang_rt.tsan_osx_dynamic.dylib:arm64+0x35db8)
    #1 std::sys::pal::unix::sync::mutex::Mutex::init::h5ed2ad5ecd6004b9 <null> (test_gc-8186109d9721178d:arm64+0x1000b8ec4)

  Mutex M1 (0x000105601d00) created at:
    #0 pthread_mutex_init <null> (libclang_rt.tsan_osx_dynamic.dylib:arm64+0x35db8)
    #1 std::sys::pal::unix::sync::mutex::Mutex::init::h5ed2ad5ecd6004b9 <null> (test_gc-8186109d9721178d:arm64+0x1000b8ec4)

  Thread T3 (tid=53510755, running) created by main thread at:
    #0 pthread_create <null> (libclang_rt.tsan_osx_dynamic.dylib:arm64+0x34418)
    #1 std::sys::pal::unix::thread::Thread::new::h95c43de68cf4a977 <null> (test_gc-8186109d9721178d:arm64+0x1000b90f0)

  Thread T4 (tid=53510756, running) created by main thread at:
    #0 pthread_create <null> (libclang_rt.tsan_osx_dynamic.dylib:arm64+0x34418)
    #1 std::sys::pal::unix::thread::Thread::new::h95c43de68cf4a977 <null> (test_gc-8186109d9721178d:arm64+0x1000b90f0)

SUMMARY: ThreadSanitizer: data race objimpl.h:24 in _PyObject_VAR_SIZE
==================

==================
WARNING: ThreadSanitizer: data race (pid=82689)
  Read of size 8 at 0x000114040048 by thread T3 (mutexes: write M0):
    #0 _PyObject_VAR_SIZE objimpl.h:25 (libpython3.14td.dylib:arm64+0x2c7668)
    #1 _PyType_AllocNoTrack typeobject.c:2293 (libpython3.14td.dylib:arm64+0x2c7414)
    #2 PyType_GenericAlloc typeobject.c:2331 (libpython3.14td.dylib:arm64+0x2c7360)
    #3 _$LT$pyo3..impl_..pyclass_init..PyNativeTypeInitializer$LT$T$GT$$u20$as$u20$pyo3..impl_..pyclass_init..PyObjectInit$LT$T$GT$$GT$::into_new_object::inner::h607105330924cb56 <null> (test_gc-8186109d9721178d:arm64+0x10008dbb0)

  Previous write of size 8 at 0x000114040048 by thread T4 (mutexes: write M1):
    #0 inherit_special typeobject.c:8029 (libpython3.14td.dylib:arm64+0x2ead78)
    #1 type_ready_inherit typeobject.c:8579 (libpython3.14td.dylib:arm64+0x2e93f4)
    #2 type_ready typeobject.c:8812 (libpython3.14td.dylib:arm64+0x2d1d3c)
    #3 PyType_Ready typeobject.c:8865 (libpython3.14td.dylib:arm64+0x2cb510)
    #4 PyType_FromMetaclass typeobject.c:5239 (libpython3.14td.dylib:arm64+0x2c9a14)
    #5 PyType_FromSpec typeobject.c:5326 (libpython3.14td.dylib:arm64+0x2cbb68)
    #6 pyo3::pyclass::create_type_object::PyTypeBuilder::build::hf1a96bf6ff0a27e0 <null> (test_gc-8186109d9721178d:arm64+0x10007e660)

  Mutex M0 (0x000105601dc0) created at:
    #0 pthread_mutex_init <null> (libclang_rt.tsan_osx_dynamic.dylib:arm64+0x35db8)
    #1 std::sys::pal::unix::sync::mutex::Mutex::init::h5ed2ad5ecd6004b9 <null> (test_gc-8186109d9721178d:arm64+0x1000b8ec4)

  Mutex M1 (0x000105601d00) created at:
    #0 pthread_mutex_init <null> (libclang_rt.tsan_osx_dynamic.dylib:arm64+0x35db8)
    #1 std::sys::pal::unix::sync::mutex::Mutex::init::h5ed2ad5ecd6004b9 <null> (test_gc-8186109d9721178d:arm64+0x1000b8ec4)

  Thread T3 (tid=53510755, running) created by main thread at:
    #0 pthread_create <null> (libclang_rt.tsan_osx_dynamic.dylib:arm64+0x34418)
    #1 std::sys::pal::unix::thread::Thread::new::h95c43de68cf4a977 <null> (test_gc-8186109d9721178d:arm64+0x1000b90f0)

  Thread T4 (tid=53510756, running) created by main thread at:
    #0 pthread_create <null> (libclang_rt.tsan_osx_dynamic.dylib:arm64+0x34418)
    #1 std::sys::pal::unix::thread::Thread::new::h95c43de68cf4a977 <null> (test_gc-8186109d9721178d:arm64+0x1000b90f0)

SUMMARY: ThreadSanitizer: data race objimpl.h:25 in _PyObject_VAR_SIZE

WARNING: ThreadSanitizer: data race (pid=83030)
  Read of size 8 at 0x00010e0408c8 by thread T9 (mutexes: write M0):
    #0 _Py_INCREF_TYPE pycore_object.h:383 (libpython3.14td.dylib:arm64+0x2d5250)
    #1 _PyObject_Init pycore_object.h:512 (libpython3.14td.dylib:arm64+0x2c79ac)
    #2 _PyType_AllocNoTrack typeobject.c:2317 (libpython3.14td.dylib:arm64+0x2c758c)
    #3 PyType_GenericAlloc typeobject.c:2331 (libpython3.14td.dylib:arm64+0x2c7360)
    #4 _$LT$pyo3..impl_..pyclass_init..PyNativeTypeInitializer$LT$T$GT$$u20$as$u20$pyo3..impl_..pyclass_init..PyObjectInit$LT$T$GT$$GT$::into_new_object::inner::h607105330924cb56 <null> (test_gc-8186109d9721178d:arm64+0x10008dbb0)

  Previous write of size 8 at 0x00010e0408c8 by thread T13 (mutexes: write M1):
    #0 PyType_FromMetaclass typeobject.c:5230 (libpython3.14td.dylib:arm64+0x2c9a00)
    #1 PyType_FromSpec typeobject.c:5326 (libpython3.14td.dylib:arm64+0x2cbb68)
    #2 pyo3::pyclass::create_type_object::PyTypeBuilder::build::hf1a96bf6ff0a27e0 <null> (test_gc-8186109d9721178d:arm64+0x10007e660)

  Mutex M0 (0x000107c01940) created at:
    #0 pthread_mutex_init <null> (libclang_rt.tsan_osx_dynamic.dylib:arm64+0x35db8)
    #1 std::sys::pal::unix::sync::mutex::Mutex::init::h5ed2ad5ecd6004b9 <null> (test_gc-8186109d9721178d:arm64+0x1000b8ec4)

  Mutex M1 (0x000107c017c0) created at:
    #0 pthread_mutex_init <null> (libclang_rt.tsan_osx_dynamic.dylib:arm64+0x35db8)
    #1 std::sys::pal::unix::sync::mutex::Mutex::init::h5ed2ad5ecd6004b9 <null> (test_gc-8186109d9721178d:arm64+0x1000b8ec4)

  Thread T9 (tid=53512561, running) created by main thread at:
    #0 pthread_create <null> (libclang_rt.tsan_osx_dynamic.dylib:arm64+0x34418)
    #1 std::sys::pal::unix::thread::Thread::new::h95c43de68cf4a977 <null> (test_gc-8186109d9721178d:arm64+0x1000b90f0)

  Thread T13 (tid=53512565, running) created by main thread at:
    #0 pthread_create <null> (libclang_rt.tsan_osx_dynamic.dylib:arm64+0x34418)
    #1 std::sys::pal::unix::thread::Thread::new::h95c43de68cf4a977 <null> (test_gc-8186109d9721178d:arm64+0x1000b90f0)

SUMMARY: ThreadSanitizer: data race pycore_object.h:383 in _Py_INCREF_TYPE

pyenv still needs to be updated for 3.14b1 so once that is done I can try again with the latest and greatest version.

Ping @colesbury. These are probably races in CPython but we can probably do more to make it clearer what the problem is. The code causing the race is generated code from proc macros.

May 07 '25 20:05 ngoldbaum