`crossbeam_epoch` - TSAN warnings when running multiple user-defined Rayon thread pools concurrently

Open gendx opened this issue 2 months ago • 2 comments

The following test spawns 10 local Rayon thread pools in parallel, and runs a bunch of iterator work on each of them.

#[test]
fn test_concurrent_thread_pools() {
    use rayon::iter::{IntoParallelRefIterator, ParallelIterator};

    std::thread::scope(|scope| {
        // Spawn several Rayon thread pools.
        for _ in 0..10 {
            scope.spawn(|| {
                let rayon_pool = rayon::ThreadPoolBuilder::new()
                    .num_threads(4)
                    .build()
                    .unwrap();

                // Do some work on each pool.
                rayon_pool.install(|| {
                    // Doesn't really matter. Just make sure each thread pool has enough work.
                    const INPUT_LEN: u64 = 100_000;

                    // Doesn't matter much either, but more iterations mean higher chances of
                    // TSAN warnings.
                    for _ in 0..10 {
                        let input = (0..=INPUT_LEN).collect::<Vec<u64>>();
                        let sum = input.par_iter().sum::<u64>();
                        assert_eq!(sum, INPUT_LEN * (INPUT_LEN + 1) / 2);
                    }
                });
            });
        }
    });
}

On my machine, it reliably triggers TSAN warnings about data races, all related to crossbeam_epoch (crossbeam_epoch::internal::Global::try_advance, crossbeam_epoch::internal::Global::collect, <T as crossbeam_epoch::atomic::Pointable>::drop, etc.).

System details:

$ cargo +nightly --version --verbose
cargo 1.92.0-nightly (801d9b498 2025-10-04)
release: 1.92.0-nightly
commit-hash: 801d9b4981dd07e3aecdca1ab86834c13615737e
commit-date: 2025-10-04
host: x86_64-unknown-linux-gnu

TSAN invocation:

export RUSTFLAGS="-D warnings -C debug_assertions -Zsanitizer=thread"
cargo +nightly test --release -Zbuild-std --target=x86_64-unknown-linux-gnu

Full TSAN warnings with stack traces:

running 1 test
==================
WARNING: ThreadSanitizer: data race (pid=1852202)
  Write of size 8 at 0x72740002ee00 by thread T19:
    #0 free ??:? (paralight-b12df329bbbd1876+0x34be7a) (BuildId: 632355c6cdc6733a800d61fbdb55837a82c0d6c2)
    #1 __rustc::__rdl_dealloc std.143dd04dfe943471-cgu.04:? (paralight-b12df329bbbd1876+0xed1684) (BuildId: 632355c6cdc6733a800d61fbdb55837a82c0d6c2)
    #2 <T as crossbeam_epoch::atomic::Pointable>::drop crossbeam_epoch.32bd66ef788b0d3c-cgu.0:? (paralight-b12df329bbbd1876+0xe9a3f3) (BuildId: 632355c6cdc6733a800d61fbdb55837a82c0d6c2)
    #3 crossbeam_epoch::deferred::Deferred::new::call crossbeam_epoch.32bd66ef788b0d3c-cgu.1:? (paralight-b12df329bbbd1876+0xe9b9ad) (BuildId: 632355c6cdc6733a800d61fbdb55837a82c0d6c2)
    #4 crossbeam_epoch::internal::Global::collect ??:? (paralight-b12df329bbbd1876+0xe99b94) (BuildId: 632355c6cdc6733a800d61fbdb55837a82c0d6c2)
    #5 crossbeam_epoch::default::pin::{{closure}} rayon_core.400168e8efdf748d-cgu.01:? (paralight-b12df329bbbd1876+0xe8a78e) (BuildId: 632355c6cdc6733a800d61fbdb55837a82c0d6c2)
    #6 crossbeam_epoch::default::with_handle rayon_core.400168e8efdf748d-cgu.01:? (paralight-b12df329bbbd1876+0xe8a4e7) (BuildId: 632355c6cdc6733a800d61fbdb55837a82c0d6c2)
    #7 crossbeam_deque::deque::Stealer<T>::steal rayon_core.400168e8efdf748d-cgu.01:? (paralight-b12df329bbbd1876+0xe898dd) (BuildId: 632355c6cdc6733a800d61fbdb55837a82c0d6c2)
    #8 rayon_core::registry::WorkerThread::find_work rayon_core.400168e8efdf748d-cgu.00:? (paralight-b12df329bbbd1876+0xe8297a) (BuildId: 632355c6cdc6733a800d61fbdb55837a82c0d6c2)
    #9 rayon_core::registry::WorkerThread::wait_until_cold ??:? (paralight-b12df329bbbd1876+0xe82373) (BuildId: 632355c6cdc6733a800d61fbdb55837a82c0d6c2)
    #10 rayon_core::registry::ThreadBuilder::run ??:? (paralight-b12df329bbbd1876+0xe83383) (BuildId: 632355c6cdc6733a800d61fbdb55837a82c0d6c2)
    #11 std::sys::backtrace::__rust_begin_short_backtrace ??:? (paralight-b12df329bbbd1876+0xe95daf) (BuildId: 632355c6cdc6733a800d61fbdb55837a82c0d6c2)
    #12 std::panicking::catch_unwind::do_call rayon_core.400168e8efdf748d-cgu.05:? (paralight-b12df329bbbd1876+0xe9190c) (BuildId: 632355c6cdc6733a800d61fbdb55837a82c0d6c2)
    #13 __rust_try.llvm.1073886105133930615 rayon_core.400168e8efdf748d-cgu.05:? (paralight-b12df329bbbd1876+0xe93921) (BuildId: 632355c6cdc6733a800d61fbdb55837a82c0d6c2)
    #14 core::ops::function::FnOnce::call_once{{vtable.shim}} rayon_core.400168e8efdf748d-cgu.05:? (paralight-b12df329bbbd1876+0xe91ad4) (BuildId: 632355c6cdc6733a800d61fbdb55837a82c0d6c2)
    #15 <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once std.143dd04dfe943471-cgu.09:? (paralight-b12df329bbbd1876+0xf0c788) (BuildId: 632355c6cdc6733a800d61fbdb55837a82c0d6c2)
    #16 std::sys::thread::unix::Thread::new::thread_start std.143dd04dfe943471-cgu.15:? (paralight-b12df329bbbd1876+0xf50e49) (BuildId: 632355c6cdc6733a800d61fbdb55837a82c0d6c2)

  Previous atomic read of size 8 at 0x72740002ee00 by thread T35:
    #0 crossbeam_epoch::internal::Global::try_advance crossbeam_epoch.32bd66ef788b0d3c-cgu.0:? (paralight-b12df329bbbd1876+0xe99705) (BuildId: 632355c6cdc6733a800d61fbdb55837a82c0d6c2)
    #1 crossbeam_epoch::internal::Global::collect ??:? (paralight-b12df329bbbd1876+0xe9999e) (BuildId: 632355c6cdc6733a800d61fbdb55837a82c0d6c2)
    #2 crossbeam_epoch::default::pin::{{closure}} rayon_core.400168e8efdf748d-cgu.01:? (paralight-b12df329bbbd1876+0xe8a78e) (BuildId: 632355c6cdc6733a800d61fbdb55837a82c0d6c2)
    #3 crossbeam_epoch::default::with_handle rayon_core.400168e8efdf748d-cgu.01:? (paralight-b12df329bbbd1876+0xe8a4e7) (BuildId: 632355c6cdc6733a800d61fbdb55837a82c0d6c2)
    #4 crossbeam_deque::deque::Stealer<T>::steal rayon_core.400168e8efdf748d-cgu.01:? (paralight-b12df329bbbd1876+0xe898dd) (BuildId: 632355c6cdc6733a800d61fbdb55837a82c0d6c2)
    #5 core::ops::function::impls::<impl core::ops::function::FnMut<A> for &mut F>::call_mut rayon_core.400168e8efdf748d-cgu.07:? (paralight-b12df329bbbd1876+0xe95f97) (BuildId: 632355c6cdc6733a800d61fbdb55837a82c0d6c2)
    #6 <core::iter::adapters::chain::Chain<A,B> as core::iter::traits::iterator::Iterator>::try_fold rayon_core.400168e8efdf748d-cgu.03:? (paralight-b12df329bbbd1876+0xe8dc91) (BuildId: 632355c6cdc6733a800d61fbdb55837a82c0d6c2)
    #7 rayon_core::registry::WorkerThread::find_work rayon_core.400168e8efdf748d-cgu.00:? (paralight-b12df329bbbd1876+0xe82bf2) (BuildId: 632355c6cdc6733a800d61fbdb55837a82c0d6c2)
    #8 rayon_core::registry::WorkerThread::wait_until_cold ??:? (paralight-b12df329bbbd1876+0xe82373) (BuildId: 632355c6cdc6733a800d61fbdb55837a82c0d6c2)
    #9 rayon_core::registry::ThreadBuilder::run ??:? (paralight-b12df329bbbd1876+0xe83383) (BuildId: 632355c6cdc6733a800d61fbdb55837a82c0d6c2)
    #10 std::sys::backtrace::__rust_begin_short_backtrace ??:? (paralight-b12df329bbbd1876+0xe95daf) (BuildId: 632355c6cdc6733a800d61fbdb55837a82c0d6c2)
    #11 std::panicking::catch_unwind::do_call rayon_core.400168e8efdf748d-cgu.05:? (paralight-b12df329bbbd1876+0xe9190c) (BuildId: 632355c6cdc6733a800d61fbdb55837a82c0d6c2)
    #12 __rust_try.llvm.1073886105133930615 rayon_core.400168e8efdf748d-cgu.05:? (paralight-b12df329bbbd1876+0xe93921) (BuildId: 632355c6cdc6733a800d61fbdb55837a82c0d6c2)
    #13 core::ops::function::FnOnce::call_once{{vtable.shim}} rayon_core.400168e8efdf748d-cgu.05:? (paralight-b12df329bbbd1876+0xe91ad4) (BuildId: 632355c6cdc6733a800d61fbdb55837a82c0d6c2)
    #14 <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once std.143dd04dfe943471-cgu.09:? (paralight-b12df329bbbd1876+0xf0c788) (BuildId: 632355c6cdc6733a800d61fbdb55837a82c0d6c2)
    #15 std::sys::thread::unix::Thread::new::thread_start std.143dd04dfe943471-cgu.15:? (paralight-b12df329bbbd1876+0xf50e49) (BuildId: 632355c6cdc6733a800d61fbdb55837a82c0d6c2)

  Thread T19 (tid=1852222, running) created by thread T11 at:
    #0 pthread_create ??:? (paralight-b12df329bbbd1876+0x34d22a) (BuildId: 632355c6cdc6733a800d61fbdb55837a82c0d6c2)
    #1 std::sys::thread::unix::Thread::new ??:? (paralight-b12df329bbbd1876+0xf51394) (BuildId: 632355c6cdc6733a800d61fbdb55837a82c0d6c2)
    #2 std::thread::Builder::spawn_unchecked rayon_core.400168e8efdf748d-cgu.05:? (paralight-b12df329bbbd1876+0xe91109) (BuildId: 632355c6cdc6733a800d61fbdb55837a82c0d6c2)
    #3 <rayon_core::registry::DefaultSpawn as rayon_core::registry::ThreadSpawn>::spawn ??:? (paralight-b12df329bbbd1876+0xe88113) (BuildId: 632355c6cdc6733a800d61fbdb55837a82c0d6c2)
    #4 rayon_core::registry::Registry::new paralight.ca2604a9752bae31-cgu.04:? (paralight-b12df329bbbd1876+0x76988d) (BuildId: 632355c6cdc6733a800d61fbdb55837a82c0d6c2)
    #5 rayon_core::thread_pool::ThreadPool::build paralight.ca2604a9752bae31-cgu.15:? (paralight-b12df329bbbd1876+0xb999a3) (BuildId: 632355c6cdc6733a800d61fbdb55837a82c0d6c2)
    #6 std::sys::backtrace::__rust_begin_short_backtrace paralight.ca2604a9752bae31-cgu.14:? (paralight-b12df329bbbd1876+0xb84c43) (BuildId: 632355c6cdc6733a800d61fbdb55837a82c0d6c2)
    #7 std::panicking::catch_unwind::do_call paralight.ca2604a9752bae31-cgu.15:? (paralight-b12df329bbbd1876+0xba5bff) (BuildId: 632355c6cdc6733a800d61fbdb55837a82c0d6c2)
    #8 __rust_try.llvm.2012152785340314153 paralight.ca2604a9752bae31-cgu.15:? (paralight-b12df329bbbd1876+0xbb6371) (BuildId: 632355c6cdc6733a800d61fbdb55837a82c0d6c2)
    #9 core::ops::function::FnOnce::call_once{{vtable.shim}} paralight.ca2604a9752bae31-cgu.15:? (paralight-b12df329bbbd1876+0xba7871) (BuildId: 632355c6cdc6733a800d61fbdb55837a82c0d6c2)
    #10 <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once std.143dd04dfe943471-cgu.09:? (paralight-b12df329bbbd1876+0xf0c788) (BuildId: 632355c6cdc6733a800d61fbdb55837a82c0d6c2)
    #11 std::sys::thread::unix::Thread::new::thread_start std.143dd04dfe943471-cgu.15:? (paralight-b12df329bbbd1876+0xf50e49) (BuildId: 632355c6cdc6733a800d61fbdb55837a82c0d6c2)

  Thread T35 (tid=1852238, running) created by thread T24 at:
    #0 pthread_create ??:? (paralight-b12df329bbbd1876+0x34d22a) (BuildId: 632355c6cdc6733a800d61fbdb55837a82c0d6c2)
    #1 std::sys::thread::unix::Thread::new ??:? (paralight-b12df329bbbd1876+0xf51394) (BuildId: 632355c6cdc6733a800d61fbdb55837a82c0d6c2)
    #2 std::thread::Builder::spawn_unchecked rayon_core.400168e8efdf748d-cgu.05:? (paralight-b12df329bbbd1876+0xe91109) (BuildId: 632355c6cdc6733a800d61fbdb55837a82c0d6c2)
    #3 <rayon_core::registry::DefaultSpawn as rayon_core::registry::ThreadSpawn>::spawn ??:? (paralight-b12df329bbbd1876+0xe88113) (BuildId: 632355c6cdc6733a800d61fbdb55837a82c0d6c2)
    #4 rayon_core::registry::Registry::new paralight.ca2604a9752bae31-cgu.04:? (paralight-b12df329bbbd1876+0x76988d) (BuildId: 632355c6cdc6733a800d61fbdb55837a82c0d6c2)
    #5 rayon_core::thread_pool::ThreadPool::build paralight.ca2604a9752bae31-cgu.15:? (paralight-b12df329bbbd1876+0xb999a3) (BuildId: 632355c6cdc6733a800d61fbdb55837a82c0d6c2)
    #6 std::sys::backtrace::__rust_begin_short_backtrace paralight.ca2604a9752bae31-cgu.14:? (paralight-b12df329bbbd1876+0xb84c43) (BuildId: 632355c6cdc6733a800d61fbdb55837a82c0d6c2)
    #7 std::panicking::catch_unwind::do_call paralight.ca2604a9752bae31-cgu.15:? (paralight-b12df329bbbd1876+0xba5bff) (BuildId: 632355c6cdc6733a800d61fbdb55837a82c0d6c2)
    #8 __rust_try.llvm.2012152785340314153 paralight.ca2604a9752bae31-cgu.15:? (paralight-b12df329bbbd1876+0xbb6371) (BuildId: 632355c6cdc6733a800d61fbdb55837a82c0d6c2)
    #9 core::ops::function::FnOnce::call_once{{vtable.shim}} paralight.ca2604a9752bae31-cgu.15:? (paralight-b12df329bbbd1876+0xba7871) (BuildId: 632355c6cdc6733a800d61fbdb55837a82c0d6c2)
    #10 <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once std.143dd04dfe943471-cgu.09:? (paralight-b12df329bbbd1876+0xf0c788) (BuildId: 632355c6cdc6733a800d61fbdb55837a82c0d6c2)
    #11 std::sys::thread::unix::Thread::new::thread_start std.143dd04dfe943471-cgu.15:? (paralight-b12df329bbbd1876+0xf50e49) (BuildId: 632355c6cdc6733a800d61fbdb55837a82c0d6c2)

SUMMARY: ThreadSanitizer: data race ??:? in free
==================
==================
WARNING: ThreadSanitizer: data race (pid=1852202)
  Write of size 8 at 0x7274001ef000 by thread T8:
    #0 free ??:? (paralight-b12df329bbbd1876+0x34be7a) (BuildId: 632355c6cdc6733a800d61fbdb55837a82c0d6c2)
    #1 __rustc::__rdl_dealloc std.143dd04dfe943471-cgu.04:? (paralight-b12df329bbbd1876+0xed1684) (BuildId: 632355c6cdc6733a800d61fbdb55837a82c0d6c2)
    #2 <T as crossbeam_epoch::atomic::Pointable>::drop crossbeam_epoch.32bd66ef788b0d3c-cgu.0:? (paralight-b12df329bbbd1876+0xe9a663) (BuildId: 632355c6cdc6733a800d61fbdb55837a82c0d6c2)
    #3 crossbeam_epoch::deferred::Deferred::new::call crossbeam_epoch.32bd66ef788b0d3c-cgu.1:? (paralight-b12df329bbbd1876+0xe9bb1d) (BuildId: 632355c6cdc6733a800d61fbdb55837a82c0d6c2)
    #4 crossbeam_epoch::internal::Global::collect ??:? (paralight-b12df329bbbd1876+0xe99b94) (BuildId: 632355c6cdc6733a800d61fbdb55837a82c0d6c2)
    #5 crossbeam_epoch::default::pin::{{closure}} rayon_core.400168e8efdf748d-cgu.01:? (paralight-b12df329bbbd1876+0xe8a78e) (BuildId: 632355c6cdc6733a800d61fbdb55837a82c0d6c2)
    #6 crossbeam_epoch::default::with_handle rayon_core.400168e8efdf748d-cgu.01:? (paralight-b12df329bbbd1876+0xe8a4e7) (BuildId: 632355c6cdc6733a800d61fbdb55837a82c0d6c2)
    #7 crossbeam_deque::deque::Stealer<T>::steal rayon_core.400168e8efdf748d-cgu.01:? (paralight-b12df329bbbd1876+0xe898dd) (BuildId: 632355c6cdc6733a800d61fbdb55837a82c0d6c2)
    #8 rayon_core::registry::WorkerThread::find_work rayon_core.400168e8efdf748d-cgu.00:? (paralight-b12df329bbbd1876+0xe8297a) (BuildId: 632355c6cdc6733a800d61fbdb55837a82c0d6c2)
    #9 rayon_core::registry::WorkerThread::wait_until_cold ??:? (paralight-b12df329bbbd1876+0xe82373) (BuildId: 632355c6cdc6733a800d61fbdb55837a82c0d6c2)
    #10 rayon_core::join::join_context::{{closure}} paralight.ca2604a9752bae31-cgu.04:? (paralight-b12df329bbbd1876+0x707d2b) (BuildId: 632355c6cdc6733a800d61fbdb55837a82c0d6c2)
    #11 rayon_core::registry::in_worker paralight.ca2604a9752bae31-cgu.04:? (paralight-b12df329bbbd1876+0x79f88d) (BuildId: 632355c6cdc6733a800d61fbdb55837a82c0d6c2)
    #12 rayon::iter::plumbing::bridge_producer_consumer::helper paralight.ca2604a9752bae31-cgu.12:? (paralight-b12df329bbbd1876+0xb673ab) (BuildId: 632355c6cdc6733a800d61fbdb55837a82c0d6c2)
    #13 std::panicking::catch_unwind::do_call paralight.ca2604a9752bae31-cgu.00:? (paralight-b12df329bbbd1876+0x4696f3) (BuildId: 632355c6cdc6733a800d61fbdb55837a82c0d6c2)
    #14 __rust_try paralight.ca2604a9752bae31-cgu.00:? (paralight-b12df329bbbd1876+0x595631) (BuildId: 632355c6cdc6733a800d61fbdb55837a82c0d6c2)
    #15 <rayon_core::job::StackJob<L,F,R> as rayon_core::job::Job>::execute paralight.ca2604a9752bae31-cgu.00:? (paralight-b12df329bbbd1876+0x55ca55) (BuildId: 632355c6cdc6733a800d61fbdb55837a82c0d6c2)
    #16 rayon_core::registry::WorkerThread::wait_until_cold ??:? (paralight-b12df329bbbd1876+0xe824c9) (BuildId: 632355c6cdc6733a800d61fbdb55837a82c0d6c2)
    #17 rayon_core::join::join_context::{{closure}} paralight.ca2604a9752bae31-cgu.04:? (paralight-b12df329bbbd1876+0x707d2b) (BuildId: 632355c6cdc6733a800d61fbdb55837a82c0d6c2)
    #18 rayon_core::registry::in_worker paralight.ca2604a9752bae31-cgu.04:? (paralight-b12df329bbbd1876+0x79f88d) (BuildId: 632355c6cdc6733a800d61fbdb55837a82c0d6c2)
    #19 rayon::iter::plumbing::bridge_producer_consumer::helper paralight.ca2604a9752bae31-cgu.12:? (paralight-b12df329bbbd1876+0xb673ab) (BuildId: 632355c6cdc6733a800d61fbdb55837a82c0d6c2)
    #20 std::panicking::catch_unwind::do_call paralight.ca2604a9752bae31-cgu.00:? (paralight-b12df329bbbd1876+0x4696f3) (BuildId: 632355c6cdc6733a800d61fbdb55837a82c0d6c2)
    #21 __rust_try paralight.ca2604a9752bae31-cgu.00:? (paralight-b12df329bbbd1876+0x595631) (BuildId: 632355c6cdc6733a800d61fbdb55837a82c0d6c2)
    #22 <rayon_core::job::StackJob<L,F,R> as rayon_core::job::Job>::execute paralight.ca2604a9752bae31-cgu.00:? (paralight-b12df329bbbd1876+0x55ca55) (BuildId: 632355c6cdc6733a800d61fbdb55837a82c0d6c2)
    #23 rayon_core::registry::WorkerThread::wait_until_cold ??:? (paralight-b12df329bbbd1876+0xe824c9) (BuildId: 632355c6cdc6733a800d61fbdb55837a82c0d6c2)
    #24 rayon_core::join::join_context::{{closure}} paralight.ca2604a9752bae31-cgu.04:? (paralight-b12df329bbbd1876+0x707d2b) (BuildId: 632355c6cdc6733a800d61fbdb55837a82c0d6c2)
    #25 rayon_core::registry::in_worker paralight.ca2604a9752bae31-cgu.04:? (paralight-b12df329bbbd1876+0x79f88d) (BuildId: 632355c6cdc6733a800d61fbdb55837a82c0d6c2)
    #26 rayon::iter::plumbing::bridge_producer_consumer::helper paralight.ca2604a9752bae31-cgu.12:? (paralight-b12df329bbbd1876+0xb673ab) (BuildId: 632355c6cdc6733a800d61fbdb55837a82c0d6c2)
    #27 std::panicking::catch_unwind::do_call paralight.ca2604a9752bae31-cgu.00:? (paralight-b12df329bbbd1876+0x4696f3) (BuildId: 632355c6cdc6733a800d61fbdb55837a82c0d6c2)
    #28 __rust_try paralight.ca2604a9752bae31-cgu.00:? (paralight-b12df329bbbd1876+0x595631) (BuildId: 632355c6cdc6733a800d61fbdb55837a82c0d6c2)
    #29 <rayon_core::job::StackJob<L,F,R> as rayon_core::job::Job>::execute paralight.ca2604a9752bae31-cgu.00:? (paralight-b12df329bbbd1876+0x55ca55) (BuildId: 632355c6cdc6733a800d61fbdb55837a82c0d6c2)
    #30 rayon_core::registry::WorkerThread::wait_until_cold ??:? (paralight-b12df329bbbd1876+0xe824c9) (BuildId: 632355c6cdc6733a800d61fbdb55837a82c0d6c2)
    #31 rayon_core::join::join_context::{{closure}} paralight.ca2604a9752bae31-cgu.04:? (paralight-b12df329bbbd1876+0x707d2b) (BuildId: 632355c6cdc6733a800d61fbdb55837a82c0d6c2)
    #32 rayon_core::registry::in_worker paralight.ca2604a9752bae31-cgu.04:? (paralight-b12df329bbbd1876+0x79f88d) (BuildId: 632355c6cdc6733a800d61fbdb55837a82c0d6c2)
    #33 rayon::iter::plumbing::bridge_producer_consumer::helper paralight.ca2604a9752bae31-cgu.12:? (paralight-b12df329bbbd1876+0xb673ab) (BuildId: 632355c6cdc6733a800d61fbdb55837a82c0d6c2)
    #34 std::panicking::catch_unwind::do_call paralight.ca2604a9752bae31-cgu.04:? (paralight-b12df329bbbd1876+0x7cf72b) (BuildId: 632355c6cdc6733a800d61fbdb55837a82c0d6c2)
    #35 __rust_try paralight.ca2604a9752bae31-cgu.04:? (paralight-b12df329bbbd1876+0x7d1289) (BuildId: 632355c6cdc6733a800d61fbdb55837a82c0d6c2)
    #36 rayon_core::join::join_context::{{closure}} paralight.ca2604a9752bae31-cgu.04:? (paralight-b12df329bbbd1876+0x707a47) (BuildId: 632355c6cdc6733a800d61fbdb55837a82c0d6c2)
    #37 rayon_core::registry::in_worker paralight.ca2604a9752bae31-cgu.04:? (paralight-b12df329bbbd1876+0x79f88d) (BuildId: 632355c6cdc6733a800d61fbdb55837a82c0d6c2)
    #38 rayon::iter::plumbing::bridge_producer_consumer::helper paralight.ca2604a9752bae31-cgu.12:? (paralight-b12df329bbbd1876+0xb673ab) (BuildId: 632355c6cdc6733a800d61fbdb55837a82c0d6c2)
    #39 std::panicking::catch_unwind::do_call paralight.ca2604a9752bae31-cgu.04:? (paralight-b12df329bbbd1876+0x7cf72b) (BuildId: 632355c6cdc6733a800d61fbdb55837a82c0d6c2)
    #40 __rust_try paralight.ca2604a9752bae31-cgu.04:? (paralight-b12df329bbbd1876+0x7d1289) (BuildId: 632355c6cdc6733a800d61fbdb55837a82c0d6c2)
    #41 rayon_core::join::join_context::{{closure}} paralight.ca2604a9752bae31-cgu.04:? (paralight-b12df329bbbd1876+0x707a47) (BuildId: 632355c6cdc6733a800d61fbdb55837a82c0d6c2)
    #42 rayon_core::registry::in_worker paralight.ca2604a9752bae31-cgu.04:? (paralight-b12df329bbbd1876+0x79f88d) (BuildId: 632355c6cdc6733a800d61fbdb55837a82c0d6c2)
    #43 rayon::iter::plumbing::bridge_producer_consumer::helper paralight.ca2604a9752bae31-cgu.12:? (paralight-b12df329bbbd1876+0xb673ab) (BuildId: 632355c6cdc6733a800d61fbdb55837a82c0d6c2)
    #44 rayon_core::thread_pool::ThreadPool::install::{{closure}} paralight.ca2604a9752bae31-cgu.00:? (paralight-b12df329bbbd1876+0x3d9fad) (BuildId: 632355c6cdc6733a800d61fbdb55837a82c0d6c2)
    #45 std::panicking::catch_unwind::do_call paralight.ca2604a9752bae31-cgu.00:? (paralight-b12df329bbbd1876+0x3ff2ca) (BuildId: 632355c6cdc6733a800d61fbdb55837a82c0d6c2)
    #46 std::panicking::catch_unwind::do_call paralight.ca2604a9752bae31-cgu.00:? (paralight-b12df329bbbd1876+0x3ff2ca) (BuildId: 632355c6cdc6733a800d61fbdb55837a82c0d6c2)
    #47 __rust_try paralight.ca2604a9752bae31-cgu.00:? (paralight-b12df329bbbd1876+0x595631) (BuildId: 632355c6cdc6733a800d61fbdb55837a82c0d6c2)
    #48 <rayon_core::job::StackJob<L,F,R> as rayon_core::job::Job>::execute paralight.ca2604a9752bae31-cgu.00:? (paralight-b12df329bbbd1876+0x4ddc3e) (BuildId: 632355c6cdc6733a800d61fbdb55837a82c0d6c2)
    #49 rayon_core::registry::WorkerThread::wait_until_cold ??:? (paralight-b12df329bbbd1876+0xe824c9) (BuildId: 632355c6cdc6733a800d61fbdb55837a82c0d6c2)
    #50 rayon_core::registry::ThreadBuilder::run ??:? (paralight-b12df329bbbd1876+0xe83383) (BuildId: 632355c6cdc6733a800d61fbdb55837a82c0d6c2)
    #51 std::sys::backtrace::__rust_begin_short_backtrace ??:? (paralight-b12df329bbbd1876+0xe95daf) (BuildId: 632355c6cdc6733a800d61fbdb55837a82c0d6c2)
    #52 std::panicking::catch_unwind::do_call rayon_core.400168e8efdf748d-cgu.05:? (paralight-b12df329bbbd1876+0xe9190c) (BuildId: 632355c6cdc6733a800d61fbdb55837a82c0d6c2)
    #53 __rust_try.llvm.1073886105133930615 rayon_core.400168e8efdf748d-cgu.05:? (paralight-b12df329bbbd1876+0xe93921) (BuildId: 632355c6cdc6733a800d61fbdb55837a82c0d6c2)
    #54 core::ops::function::FnOnce::call_once{{vtable.shim}} rayon_core.400168e8efdf748d-cgu.05:? (paralight-b12df329bbbd1876+0xe91ad4) (BuildId: 632355c6cdc6733a800d61fbdb55837a82c0d6c2)
    #55 <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once std.143dd04dfe943471-cgu.09:? (paralight-b12df329bbbd1876+0xf0c788) (BuildId: 632355c6cdc6733a800d61fbdb55837a82c0d6c2)
    #56 std::sys::thread::unix::Thread::new::thread_start std.143dd04dfe943471-cgu.15:? (paralight-b12df329bbbd1876+0xf50e49) (BuildId: 632355c6cdc6733a800d61fbdb55837a82c0d6c2)

  Previous read of size 8 at 0x7274001ef000 by thread T19:
    #0 crossbeam_epoch::internal::Global::collect ??:? (paralight-b12df329bbbd1876+0xe99ace) (BuildId: 632355c6cdc6733a800d61fbdb55837a82c0d6c2)
    #1 crossbeam_epoch::default::pin::{{closure}} rayon_core.400168e8efdf748d-cgu.01:? (paralight-b12df329bbbd1876+0xe8a78e) (BuildId: 632355c6cdc6733a800d61fbdb55837a82c0d6c2)
    #2 crossbeam_epoch::default::with_handle rayon_core.400168e8efdf748d-cgu.01:? (paralight-b12df329bbbd1876+0xe8a4e7) (BuildId: 632355c6cdc6733a800d61fbdb55837a82c0d6c2)
    #3 crossbeam_deque::deque::Stealer<T>::steal rayon_core.400168e8efdf748d-cgu.01:? (paralight-b12df329bbbd1876+0xe898dd) (BuildId: 632355c6cdc6733a800d61fbdb55837a82c0d6c2)
    #4 rayon_core::registry::WorkerThread::find_work rayon_core.400168e8efdf748d-cgu.00:? (paralight-b12df329bbbd1876+0xe8297a) (BuildId: 632355c6cdc6733a800d61fbdb55837a82c0d6c2)
    #5 rayon_core::registry::WorkerThread::wait_until_cold ??:? (paralight-b12df329bbbd1876+0xe82373) (BuildId: 632355c6cdc6733a800d61fbdb55837a82c0d6c2)
    #6 rayon_core::registry::ThreadBuilder::run ??:? (paralight-b12df329bbbd1876+0xe83383) (BuildId: 632355c6cdc6733a800d61fbdb55837a82c0d6c2)
    #7 std::sys::backtrace::__rust_begin_short_backtrace ??:? (paralight-b12df329bbbd1876+0xe95daf) (BuildId: 632355c6cdc6733a800d61fbdb55837a82c0d6c2)
    #8 std::panicking::catch_unwind::do_call rayon_core.400168e8efdf748d-cgu.05:? (paralight-b12df329bbbd1876+0xe9190c) (BuildId: 632355c6cdc6733a800d61fbdb55837a82c0d6c2)
    #9 __rust_try.llvm.1073886105133930615 rayon_core.400168e8efdf748d-cgu.05:? (paralight-b12df329bbbd1876+0xe93921) (BuildId: 632355c6cdc6733a800d61fbdb55837a82c0d6c2)
    #10 core::ops::function::FnOnce::call_once{{vtable.shim}} rayon_core.400168e8efdf748d-cgu.05:? (paralight-b12df329bbbd1876+0xe91ad4) (BuildId: 632355c6cdc6733a800d61fbdb55837a82c0d6c2)
    #11 <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once std.143dd04dfe943471-cgu.09:? (paralight-b12df329bbbd1876+0xf0c788) (BuildId: 632355c6cdc6733a800d61fbdb55837a82c0d6c2)
    #12 std::sys::thread::unix::Thread::new::thread_start std.143dd04dfe943471-cgu.15:? (paralight-b12df329bbbd1876+0xf50e49) (BuildId: 632355c6cdc6733a800d61fbdb55837a82c0d6c2)

  Thread T8 (tid=1852211, running) created by thread T5 at:
    #0 pthread_create ??:? (paralight-b12df329bbbd1876+0x34d22a) (BuildId: 632355c6cdc6733a800d61fbdb55837a82c0d6c2)
    #1 std::sys::thread::unix::Thread::new ??:? (paralight-b12df329bbbd1876+0xf51394) (BuildId: 632355c6cdc6733a800d61fbdb55837a82c0d6c2)
    #2 std::thread::Builder::spawn_unchecked rayon_core.400168e8efdf748d-cgu.05:? (paralight-b12df329bbbd1876+0xe91109) (BuildId: 632355c6cdc6733a800d61fbdb55837a82c0d6c2)
    #3 <rayon_core::registry::DefaultSpawn as rayon_core::registry::ThreadSpawn>::spawn ??:? (paralight-b12df329bbbd1876+0xe88113) (BuildId: 632355c6cdc6733a800d61fbdb55837a82c0d6c2)
    #4 rayon_core::registry::Registry::new paralight.ca2604a9752bae31-cgu.04:? (paralight-b12df329bbbd1876+0x76988d) (BuildId: 632355c6cdc6733a800d61fbdb55837a82c0d6c2)
    #5 rayon_core::thread_pool::ThreadPool::build paralight.ca2604a9752bae31-cgu.15:? (paralight-b12df329bbbd1876+0xb999a3) (BuildId: 632355c6cdc6733a800d61fbdb55837a82c0d6c2)
    #6 std::sys::backtrace::__rust_begin_short_backtrace paralight.ca2604a9752bae31-cgu.14:? (paralight-b12df329bbbd1876+0xb84c43) (BuildId: 632355c6cdc6733a800d61fbdb55837a82c0d6c2)
    #7 std::panicking::catch_unwind::do_call paralight.ca2604a9752bae31-cgu.15:? (paralight-b12df329bbbd1876+0xba5bff) (BuildId: 632355c6cdc6733a800d61fbdb55837a82c0d6c2)
    #8 __rust_try.llvm.2012152785340314153 paralight.ca2604a9752bae31-cgu.15:? (paralight-b12df329bbbd1876+0xbb6371) (BuildId: 632355c6cdc6733a800d61fbdb55837a82c0d6c2)
    #9 core::ops::function::FnOnce::call_once{{vtable.shim}} paralight.ca2604a9752bae31-cgu.15:? (paralight-b12df329bbbd1876+0xba7871) (BuildId: 632355c6cdc6733a800d61fbdb55837a82c0d6c2)
    #10 <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once std.143dd04dfe943471-cgu.09:? (paralight-b12df329bbbd1876+0xf0c788) (BuildId: 632355c6cdc6733a800d61fbdb55837a82c0d6c2)
    #11 std::sys::thread::unix::Thread::new::thread_start std.143dd04dfe943471-cgu.15:? (paralight-b12df329bbbd1876+0xf50e49) (BuildId: 632355c6cdc6733a800d61fbdb55837a82c0d6c2)

  Thread T19 (tid=1852222, running) created by thread T11 at:
    #0 pthread_create ??:? (paralight-b12df329bbbd1876+0x34d22a) (BuildId: 632355c6cdc6733a800d61fbdb55837a82c0d6c2)
    #1 std::sys::thread::unix::Thread::new ??:? (paralight-b12df329bbbd1876+0xf51394) (BuildId: 632355c6cdc6733a800d61fbdb55837a82c0d6c2)
    #2 std::thread::Builder::spawn_unchecked rayon_core.400168e8efdf748d-cgu.05:? (paralight-b12df329bbbd1876+0xe91109) (BuildId: 632355c6cdc6733a800d61fbdb55837a82c0d6c2)
    #3 <rayon_core::registry::DefaultSpawn as rayon_core::registry::ThreadSpawn>::spawn ??:? (paralight-b12df329bbbd1876+0xe88113) (BuildId: 632355c6cdc6733a800d61fbdb55837a82c0d6c2)
    #4 rayon_core::registry::Registry::new paralight.ca2604a9752bae31-cgu.04:? (paralight-b12df329bbbd1876+0x76988d) (BuildId: 632355c6cdc6733a800d61fbdb55837a82c0d6c2)
    #5 rayon_core::thread_pool::ThreadPool::build paralight.ca2604a9752bae31-cgu.15:? (paralight-b12df329bbbd1876+0xb999a3) (BuildId: 632355c6cdc6733a800d61fbdb55837a82c0d6c2)
    #6 std::sys::backtrace::__rust_begin_short_backtrace paralight.ca2604a9752bae31-cgu.14:? (paralight-b12df329bbbd1876+0xb84c43) (BuildId: 632355c6cdc6733a800d61fbdb55837a82c0d6c2)
    #7 std::panicking::catch_unwind::do_call paralight.ca2604a9752bae31-cgu.15:? (paralight-b12df329bbbd1876+0xba5bff) (BuildId: 632355c6cdc6733a800d61fbdb55837a82c0d6c2)
    #8 __rust_try.llvm.2012152785340314153 paralight.ca2604a9752bae31-cgu.15:? (paralight-b12df329bbbd1876+0xbb6371) (BuildId: 632355c6cdc6733a800d61fbdb55837a82c0d6c2)
    #9 core::ops::function::FnOnce::call_once{{vtable.shim}} paralight.ca2604a9752bae31-cgu.15:? (paralight-b12df329bbbd1876+0xba7871) (BuildId: 632355c6cdc6733a800d61fbdb55837a82c0d6c2)
    #10 <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once std.143dd04dfe943471-cgu.09:? (paralight-b12df329bbbd1876+0xf0c788) (BuildId: 632355c6cdc6733a800d61fbdb55837a82c0d6c2)
    #11 std::sys::thread::unix::Thread::new::thread_start std.143dd04dfe943471-cgu.15:? (paralight-b12df329bbbd1876+0xf50e49) (BuildId: 632355c6cdc6733a800d61fbdb55837a82c0d6c2)

SUMMARY: ThreadSanitizer: data race ??:? in free
==================
==================
WARNING: ThreadSanitizer: data race (pid=1852202)
  Write of size 8 at 0x727400180600 by thread T44:
    #0 free ??:? (paralight-b12df329bbbd1876+0x34be7a) (BuildId: 632355c6cdc6733a800d61fbdb55837a82c0d6c2)
    #1 __rustc::__rdl_dealloc std.143dd04dfe943471-cgu.04:? (paralight-b12df329bbbd1876+0xed1684) (BuildId: 632355c6cdc6733a800d61fbdb55837a82c0d6c2)
    #2 <T as crossbeam_epoch::atomic::Pointable>::drop crossbeam_epoch.32bd66ef788b0d3c-cgu.0:? (paralight-b12df329bbbd1876+0xe9a663) (BuildId: 632355c6cdc6733a800d61fbdb55837a82c0d6c2)
    #3 crossbeam_epoch::deferred::Deferred::new::call crossbeam_epoch.32bd66ef788b0d3c-cgu.1:? (paralight-b12df329bbbd1876+0xe9bb1d) (BuildId: 632355c6cdc6733a800d61fbdb55837a82c0d6c2)
    #4 crossbeam_epoch::internal::Global::collect ??:? (paralight-b12df329bbbd1876+0xe99b94) (BuildId: 632355c6cdc6733a800d61fbdb55837a82c0d6c2)
    #5 crossbeam_epoch::default::pin::{{closure}} rayon_core.400168e8efdf748d-cgu.01:? (paralight-b12df329bbbd1876+0xe8a78e) (BuildId: 632355c6cdc6733a800d61fbdb55837a82c0d6c2)
    #6 crossbeam_epoch::default::with_handle rayon_core.400168e8efdf748d-cgu.01:? (paralight-b12df329bbbd1876+0xe8a4e7) (BuildId: 632355c6cdc6733a800d61fbdb55837a82c0d6c2)
    #7 crossbeam_deque::deque::Stealer<T>::steal rayon_core.400168e8efdf748d-cgu.01:? (paralight-b12df329bbbd1876+0xe898dd) (BuildId: 632355c6cdc6733a800d61fbdb55837a82c0d6c2)
    #8 core::ops::function::impls::<impl core::ops::function::FnMut<A> for &mut F>::call_mut rayon_core.400168e8efdf748d-cgu.07:? (paralight-b12df329bbbd1876+0xe95f97) (BuildId: 632355c6cdc6733a800d61fbdb55837a82c0d6c2)
    #9 <core::iter::adapters::chain::Chain<A,B> as core::iter::traits::iterator::Iterator>::try_fold rayon_core.400168e8efdf748d-cgu.03:? (paralight-b12df329bbbd1876+0xe8dc91) (BuildId: 632355c6cdc6733a800d61fbdb55837a82c0d6c2)
    #10 rayon_core::registry::WorkerThread::find_work rayon_core.400168e8efdf748d-cgu.00:? (paralight-b12df329bbbd1876+0xe82bf2) (BuildId: 632355c6cdc6733a800d61fbdb55837a82c0d6c2)
    #11 rayon_core::registry::WorkerThread::wait_until_cold ??:? (paralight-b12df329bbbd1876+0xe82373) (BuildId: 632355c6cdc6733a800d61fbdb55837a82c0d6c2)
    #12 rayon_core::join::join_context::{{closure}} paralight.ca2604a9752bae31-cgu.04:? (paralight-b12df329bbbd1876+0x707d2b) (BuildId: 632355c6cdc6733a800d61fbdb55837a82c0d6c2)
    #13 rayon_core::registry::in_worker paralight.ca2604a9752bae31-cgu.04:? (paralight-b12df329bbbd1876+0x79f88d) (BuildId: 632355c6cdc6733a800d61fbdb55837a82c0d6c2)
    #14 rayon::iter::plumbing::bridge_producer_consumer::helper paralight.ca2604a9752bae31-cgu.12:? (paralight-b12df329bbbd1876+0xb673ab) (BuildId: 632355c6cdc6733a800d61fbdb55837a82c0d6c2)
    #15 rayon_core::job::StackJob<L,F,R>::run_inline paralight.ca2604a9752bae31-cgu.00:? (paralight-b12df329bbbd1876+0x3da3f2) (BuildId: 632355c6cdc6733a800d61fbdb55837a82c0d6c2)
    #16 rayon_core::join::join_context::{{closure}} paralight.ca2604a9752bae31-cgu.04:? (paralight-b12df329bbbd1876+0x707bb5) (BuildId: 632355c6cdc6733a800d61fbdb55837a82c0d6c2)
    #17 rayon_core::registry::in_worker paralight.ca2604a9752bae31-cgu.04:? (paralight-b12df329bbbd1876+0x79f88d) (BuildId: 632355c6cdc6733a800d61fbdb55837a82c0d6c2)
    #18 rayon::iter::plumbing::bridge_producer_consumer::helper paralight.ca2604a9752bae31-cgu.12:? (paralight-b12df329bbbd1876+0xb673ab) (BuildId: 632355c6cdc6733a800d61fbdb55837a82c0d6c2)
    #19 std::panicking::catch_unwind::do_call paralight.ca2604a9752bae31-cgu.00:? (paralight-b12df329bbbd1876+0x4696f3) (BuildId: 632355c6cdc6733a800d61fbdb55837a82c0d6c2)
    #20 __rust_try paralight.ca2604a9752bae31-cgu.00:? (paralight-b12df329bbbd1876+0x595631) (BuildId: 632355c6cdc6733a800d61fbdb55837a82c0d6c2)
    #21 <rayon_core::job::StackJob<L,F,R> as rayon_core::job::Job>::execute paralight.ca2604a9752bae31-cgu.00:? (paralight-b12df329bbbd1876+0x55ca55) (BuildId: 632355c6cdc6733a800d61fbdb55837a82c0d6c2)
    #22 rayon_core::registry::WorkerThread::wait_until_cold ??:? (paralight-b12df329bbbd1876+0xe824c9) (BuildId: 632355c6cdc6733a800d61fbdb55837a82c0d6c2)
    #23 rayon_core::registry::ThreadBuilder::run ??:? (paralight-b12df329bbbd1876+0xe83383) (BuildId: 632355c6cdc6733a800d61fbdb55837a82c0d6c2)
    #24 std::sys::backtrace::__rust_begin_short_backtrace ??:? (paralight-b12df329bbbd1876+0xe95daf) (BuildId: 632355c6cdc6733a800d61fbdb55837a82c0d6c2)
    #25 std::panicking::catch_unwind::do_call rayon_core.400168e8efdf748d-cgu.05:? (paralight-b12df329bbbd1876+0xe9190c) (BuildId: 632355c6cdc6733a800d61fbdb55837a82c0d6c2)
    #26 __rust_try.llvm.1073886105133930615 rayon_core.400168e8efdf748d-cgu.05:? (paralight-b12df329bbbd1876+0xe93921) (BuildId: 632355c6cdc6733a800d61fbdb55837a82c0d6c2)
    #27 core::ops::function::FnOnce::call_once{{vtable.shim}} rayon_core.400168e8efdf748d-cgu.05:? (paralight-b12df329bbbd1876+0xe91ad4) (BuildId: 632355c6cdc6733a800d61fbdb55837a82c0d6c2)
    #28 <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once std.143dd04dfe943471-cgu.09:? (paralight-b12df329bbbd1876+0xf0c788) (BuildId: 632355c6cdc6733a800d61fbdb55837a82c0d6c2)
    #29 std::sys::thread::unix::Thread::new::thread_start std.143dd04dfe943471-cgu.15:? (paralight-b12df329bbbd1876+0xf50e49) (BuildId: 632355c6cdc6733a800d61fbdb55837a82c0d6c2)

  Previous read of size 8 at 0x727400180600 by thread T8:
    #0 crossbeam_epoch::internal::Global::collect ??:? (paralight-b12df329bbbd1876+0xe99ace) (BuildId: 632355c6cdc6733a800d61fbdb55837a82c0d6c2)
    #1 crossbeam_epoch::default::pin::{{closure}} rayon_core.400168e8efdf748d-cgu.01:? (paralight-b12df329bbbd1876+0xe8a78e) (BuildId: 632355c6cdc6733a800d61fbdb55837a82c0d6c2)
    #2 crossbeam_epoch::default::with_handle rayon_core.400168e8efdf748d-cgu.01:? (paralight-b12df329bbbd1876+0xe8a4e7) (BuildId: 632355c6cdc6733a800d61fbdb55837a82c0d6c2)
    #3 crossbeam_deque::deque::Stealer<T>::steal rayon_core.400168e8efdf748d-cgu.01:? (paralight-b12df329bbbd1876+0xe898dd) (BuildId: 632355c6cdc6733a800d61fbdb55837a82c0d6c2)
    #4 rayon_core::registry::WorkerThread::find_work rayon_core.400168e8efdf748d-cgu.00:? (paralight-b12df329bbbd1876+0xe8297a) (BuildId: 632355c6cdc6733a800d61fbdb55837a82c0d6c2)
    #5 rayon_core::registry::WorkerThread::wait_until_cold ??:? (paralight-b12df329bbbd1876+0xe82373) (BuildId: 632355c6cdc6733a800d61fbdb55837a82c0d6c2)
    #6 rayon_core::join::join_context::{{closure}} paralight.ca2604a9752bae31-cgu.04:? (paralight-b12df329bbbd1876+0x707d2b) (BuildId: 632355c6cdc6733a800d61fbdb55837a82c0d6c2)
    #7 rayon_core::registry::in_worker paralight.ca2604a9752bae31-cgu.04:? (paralight-b12df329bbbd1876+0x79f88d) (BuildId: 632355c6cdc6733a800d61fbdb55837a82c0d6c2)
    #8 rayon::iter::plumbing::bridge_producer_consumer::helper paralight.ca2604a9752bae31-cgu.12:? (paralight-b12df329bbbd1876+0xb673ab) (BuildId: 632355c6cdc6733a800d61fbdb55837a82c0d6c2)
    #9 std::panicking::catch_unwind::do_call paralight.ca2604a9752bae31-cgu.00:? (paralight-b12df329bbbd1876+0x4696f3) (BuildId: 632355c6cdc6733a800d61fbdb55837a82c0d6c2)
    #10 __rust_try paralight.ca2604a9752bae31-cgu.00:? (paralight-b12df329bbbd1876+0x595631) (BuildId: 632355c6cdc6733a800d61fbdb55837a82c0d6c2)
    #11 <rayon_core::job::StackJob<L,F,R> as rayon_core::job::Job>::execute paralight.ca2604a9752bae31-cgu.00:? (paralight-b12df329bbbd1876+0x55ca55) (BuildId: 632355c6cdc6733a800d61fbdb55837a82c0d6c2)
    #12 rayon_core::registry::WorkerThread::wait_until_cold ??:? (paralight-b12df329bbbd1876+0xe824c9) (BuildId: 632355c6cdc6733a800d61fbdb55837a82c0d6c2)
    #13 rayon_core::join::join_context::{{closure}} paralight.ca2604a9752bae31-cgu.04:? (paralight-b12df329bbbd1876+0x707d2b) (BuildId: 632355c6cdc6733a800d61fbdb55837a82c0d6c2)
    #14 rayon_core::registry::in_worker paralight.ca2604a9752bae31-cgu.04:? (paralight-b12df329bbbd1876+0x79f88d) (BuildId: 632355c6cdc6733a800d61fbdb55837a82c0d6c2)
    #15 rayon::iter::plumbing::bridge_producer_consumer::helper paralight.ca2604a9752bae31-cgu.12:? (paralight-b12df329bbbd1876+0xb673ab) (BuildId: 632355c6cdc6733a800d61fbdb55837a82c0d6c2)
    #16 std::panicking::catch_unwind::do_call paralight.ca2604a9752bae31-cgu.00:? (paralight-b12df329bbbd1876+0x4696f3) (BuildId: 632355c6cdc6733a800d61fbdb55837a82c0d6c2)
    #17 __rust_try paralight.ca2604a9752bae31-cgu.00:? (paralight-b12df329bbbd1876+0x595631) (BuildId: 632355c6cdc6733a800d61fbdb55837a82c0d6c2)
    #18 <rayon_core::job::StackJob<L,F,R> as rayon_core::job::Job>::execute paralight.ca2604a9752bae31-cgu.00:? (paralight-b12df329bbbd1876+0x55ca55) (BuildId: 632355c6cdc6733a800d61fbdb55837a82c0d6c2)
    #19 rayon_core::registry::WorkerThread::wait_until_cold ??:? (paralight-b12df329bbbd1876+0xe824c9) (BuildId: 632355c6cdc6733a800d61fbdb55837a82c0d6c2)
    #20 rayon_core::join::join_context::{{closure}} paralight.ca2604a9752bae31-cgu.04:? (paralight-b12df329bbbd1876+0x707d2b) (BuildId: 632355c6cdc6733a800d61fbdb55837a82c0d6c2)
    #21 rayon_core::registry::in_worker paralight.ca2604a9752bae31-cgu.04:? (paralight-b12df329bbbd1876+0x79f88d) (BuildId: 632355c6cdc6733a800d61fbdb55837a82c0d6c2)
    #22 rayon::iter::plumbing::bridge_producer_consumer::helper paralight.ca2604a9752bae31-cgu.12:? (paralight-b12df329bbbd1876+0xb673ab) (BuildId: 632355c6cdc6733a800d61fbdb55837a82c0d6c2)
    #23 std::panicking::catch_unwind::do_call paralight.ca2604a9752bae31-cgu.00:? (paralight-b12df329bbbd1876+0x4696f3) (BuildId: 632355c6cdc6733a800d61fbdb55837a82c0d6c2)
    #24 __rust_try paralight.ca2604a9752bae31-cgu.00:? (paralight-b12df329bbbd1876+0x595631) (BuildId: 632355c6cdc6733a800d61fbdb55837a82c0d6c2)
    #25 <rayon_core::job::StackJob<L,F,R> as rayon_core::job::Job>::execute paralight.ca2604a9752bae31-cgu.00:? (paralight-b12df329bbbd1876+0x55ca55) (BuildId: 632355c6cdc6733a800d61fbdb55837a82c0d6c2)
    #26 rayon_core::registry::WorkerThread::wait_until_cold ??:? (paralight-b12df329bbbd1876+0xe824c9) (BuildId: 632355c6cdc6733a800d61fbdb55837a82c0d6c2)
    #27 rayon_core::join::join_context::{{closure}} paralight.ca2604a9752bae31-cgu.04:? (paralight-b12df329bbbd1876+0x707d2b) (BuildId: 632355c6cdc6733a800d61fbdb55837a82c0d6c2)
    #28 rayon_core::registry::in_worker paralight.ca2604a9752bae31-cgu.04:? (paralight-b12df329bbbd1876+0x79f88d) (BuildId: 632355c6cdc6733a800d61fbdb55837a82c0d6c2)
    #29 rayon::iter::plumbing::bridge_producer_consumer::helper paralight.ca2604a9752bae31-cgu.12:? (paralight-b12df329bbbd1876+0xb673ab) (BuildId: 632355c6cdc6733a800d61fbdb55837a82c0d6c2)
    #30 std::panicking::catch_unwind::do_call paralight.ca2604a9752bae31-cgu.04:? (paralight-b12df329bbbd1876+0x7cf72b) (BuildId: 632355c6cdc6733a800d61fbdb55837a82c0d6c2)
    #31 __rust_try paralight.ca2604a9752bae31-cgu.04:? (paralight-b12df329bbbd1876+0x7d1289) (BuildId: 632355c6cdc6733a800d61fbdb55837a82c0d6c2)
    #32 rayon_core::join::join_context::{{closure}} paralight.ca2604a9752bae31-cgu.04:? (paralight-b12df329bbbd1876+0x707a47) (BuildId: 632355c6cdc6733a800d61fbdb55837a82c0d6c2)
    #33 rayon_core::registry::in_worker paralight.ca2604a9752bae31-cgu.04:? (paralight-b12df329bbbd1876+0x79f88d) (BuildId: 632355c6cdc6733a800d61fbdb55837a82c0d6c2)
    #34 rayon::iter::plumbing::bridge_producer_consumer::helper paralight.ca2604a9752bae31-cgu.12:? (paralight-b12df329bbbd1876+0xb673ab) (BuildId: 632355c6cdc6733a800d61fbdb55837a82c0d6c2)
    #35 std::panicking::catch_unwind::do_call paralight.ca2604a9752bae31-cgu.04:? (paralight-b12df329bbbd1876+0x7cf72b) (BuildId: 632355c6cdc6733a800d61fbdb55837a82c0d6c2)
    #36 __rust_try paralight.ca2604a9752bae31-cgu.04:? (paralight-b12df329bbbd1876+0x7d1289) (BuildId: 632355c6cdc6733a800d61fbdb55837a82c0d6c2)
    #37 rayon_core::join::join_context::{{closure}} paralight.ca2604a9752bae31-cgu.04:? (paralight-b12df329bbbd1876+0x707a47) (BuildId: 632355c6cdc6733a800d61fbdb55837a82c0d6c2)
    #38 rayon_core::registry::in_worker paralight.ca2604a9752bae31-cgu.04:? (paralight-b12df329bbbd1876+0x79f88d) (BuildId: 632355c6cdc6733a800d61fbdb55837a82c0d6c2)
    #39 rayon::iter::plumbing::bridge_producer_consumer::helper paralight.ca2604a9752bae31-cgu.12:? (paralight-b12df329bbbd1876+0xb673ab) (BuildId: 632355c6cdc6733a800d61fbdb55837a82c0d6c2)
    #40 rayon_core::thread_pool::ThreadPool::install::{{closure}} paralight.ca2604a9752bae31-cgu.00:? (paralight-b12df329bbbd1876+0x3d9fad) (BuildId: 632355c6cdc6733a800d61fbdb55837a82c0d6c2)
    #41 std::panicking::catch_unwind::do_call paralight.ca2604a9752bae31-cgu.00:? (paralight-b12df329bbbd1876+0x3ff2ca) (BuildId: 632355c6cdc6733a800d61fbdb55837a82c0d6c2)
    #42 std::panicking::catch_unwind::do_call paralight.ca2604a9752bae31-cgu.00:? (paralight-b12df329bbbd1876+0x3ff2ca) (BuildId: 632355c6cdc6733a800d61fbdb55837a82c0d6c2)
    #43 __rust_try paralight.ca2604a9752bae31-cgu.00:? (paralight-b12df329bbbd1876+0x595631) (BuildId: 632355c6cdc6733a800d61fbdb55837a82c0d6c2)
    #44 <rayon_core::job::StackJob<L,F,R> as rayon_core::job::Job>::execute paralight.ca2604a9752bae31-cgu.00:? (paralight-b12df329bbbd1876+0x4ddc3e) (BuildId: 632355c6cdc6733a800d61fbdb55837a82c0d6c2)
    #45 rayon_core::registry::WorkerThread::wait_until_cold ??:? (paralight-b12df329bbbd1876+0xe824c9) (BuildId: 632355c6cdc6733a800d61fbdb55837a82c0d6c2)
    #46 rayon_core::registry::ThreadBuilder::run ??:? (paralight-b12df329bbbd1876+0xe83383) (BuildId: 632355c6cdc6733a800d61fbdb55837a82c0d6c2)
    #47 std::sys::backtrace::__rust_begin_short_backtrace ??:? (paralight-b12df329bbbd1876+0xe95daf) (BuildId: 632355c6cdc6733a800d61fbdb55837a82c0d6c2)
    #48 std::panicking::catch_unwind::do_call rayon_core.400168e8efdf748d-cgu.05:? (paralight-b12df329bbbd1876+0xe9190c) (BuildId: 632355c6cdc6733a800d61fbdb55837a82c0d6c2)
    #49 __rust_try.llvm.1073886105133930615 rayon_core.400168e8efdf748d-cgu.05:? (paralight-b12df329bbbd1876+0xe93921) (BuildId: 632355c6cdc6733a800d61fbdb55837a82c0d6c2)
    #50 core::ops::function::FnOnce::call_once{{vtable.shim}} rayon_core.400168e8efdf748d-cgu.05:? (paralight-b12df329bbbd1876+0xe91ad4) (BuildId: 632355c6cdc6733a800d61fbdb55837a82c0d6c2)
    #51 <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once std.143dd04dfe943471-cgu.09:? (paralight-b12df329bbbd1876+0xf0c788) (BuildId: 632355c6cdc6733a800d61fbdb55837a82c0d6c2)
    #52 std::sys::thread::unix::Thread::new::thread_start std.143dd04dfe943471-cgu.15:? (paralight-b12df329bbbd1876+0xf50e49) (BuildId: 632355c6cdc6733a800d61fbdb55837a82c0d6c2)

  Thread T44 (tid=1852247, running) created by thread T11 at:
    #0 pthread_create ??:? (paralight-b12df329bbbd1876+0x34d22a) (BuildId: 632355c6cdc6733a800d61fbdb55837a82c0d6c2)
    #1 std::sys::thread::unix::Thread::new ??:? (paralight-b12df329bbbd1876+0xf51394) (BuildId: 632355c6cdc6733a800d61fbdb55837a82c0d6c2)
    #2 std::thread::Builder::spawn_unchecked rayon_core.400168e8efdf748d-cgu.05:? (paralight-b12df329bbbd1876+0xe91109) (BuildId: 632355c6cdc6733a800d61fbdb55837a82c0d6c2)
    #3 <rayon_core::registry::DefaultSpawn as rayon_core::registry::ThreadSpawn>::spawn ??:? (paralight-b12df329bbbd1876+0xe88113) (BuildId: 632355c6cdc6733a800d61fbdb55837a82c0d6c2)
    #4 rayon_core::registry::Registry::new paralight.ca2604a9752bae31-cgu.04:? (paralight-b12df329bbbd1876+0x76988d) (BuildId: 632355c6cdc6733a800d61fbdb55837a82c0d6c2)
    #5 rayon_core::thread_pool::ThreadPool::build paralight.ca2604a9752bae31-cgu.15:? (paralight-b12df329bbbd1876+0xb999a3) (BuildId: 632355c6cdc6733a800d61fbdb55837a82c0d6c2)
    #6 std::sys::backtrace::__rust_begin_short_backtrace paralight.ca2604a9752bae31-cgu.14:? (paralight-b12df329bbbd1876+0xb84c43) (BuildId: 632355c6cdc6733a800d61fbdb55837a82c0d6c2)
    #7 std::panicking::catch_unwind::do_call paralight.ca2604a9752bae31-cgu.15:? (paralight-b12df329bbbd1876+0xba5bff) (BuildId: 632355c6cdc6733a800d61fbdb55837a82c0d6c2)
    #8 __rust_try.llvm.2012152785340314153 paralight.ca2604a9752bae31-cgu.15:? (paralight-b12df329bbbd1876+0xbb6371) (BuildId: 632355c6cdc6733a800d61fbdb55837a82c0d6c2)
    #9 core::ops::function::FnOnce::call_once{{vtable.shim}} paralight.ca2604a9752bae31-cgu.15:? (paralight-b12df329bbbd1876+0xba7871) (BuildId: 632355c6cdc6733a800d61fbdb55837a82c0d6c2)
    #10 <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once std.143dd04dfe943471-cgu.09:? (paralight-b12df329bbbd1876+0xf0c788) (BuildId: 632355c6cdc6733a800d61fbdb55837a82c0d6c2)
    #11 std::sys::thread::unix::Thread::new::thread_start std.143dd04dfe943471-cgu.15:? (paralight-b12df329bbbd1876+0xf50e49) (BuildId: 632355c6cdc6733a800d61fbdb55837a82c0d6c2)

  Thread T8 (tid=1852211, running) created by thread T5 at:
    #0 pthread_create ??:? (paralight-b12df329bbbd1876+0x34d22a) (BuildId: 632355c6cdc6733a800d61fbdb55837a82c0d6c2)
    #1 std::sys::thread::unix::Thread::new ??:? (paralight-b12df329bbbd1876+0xf51394) (BuildId: 632355c6cdc6733a800d61fbdb55837a82c0d6c2)
    #2 std::thread::Builder::spawn_unchecked rayon_core.400168e8efdf748d-cgu.05:? (paralight-b12df329bbbd1876+0xe91109) (BuildId: 632355c6cdc6733a800d61fbdb55837a82c0d6c2)
    #3 <rayon_core::registry::DefaultSpawn as rayon_core::registry::ThreadSpawn>::spawn ??:? (paralight-b12df329bbbd1876+0xe88113) (BuildId: 632355c6cdc6733a800d61fbdb55837a82c0d6c2)
    #4 rayon_core::registry::Registry::new paralight.ca2604a9752bae31-cgu.04:? (paralight-b12df329bbbd1876+0x76988d) (BuildId: 632355c6cdc6733a800d61fbdb55837a82c0d6c2)
    #5 rayon_core::thread_pool::ThreadPool::build paralight.ca2604a9752bae31-cgu.15:? (paralight-b12df329bbbd1876+0xb999a3) (BuildId: 632355c6cdc6733a800d61fbdb55837a82c0d6c2)
    #6 std::sys::backtrace::__rust_begin_short_backtrace paralight.ca2604a9752bae31-cgu.14:? (paralight-b12df329bbbd1876+0xb84c43) (BuildId: 632355c6cdc6733a800d61fbdb55837a82c0d6c2)
    #7 std::panicking::catch_unwind::do_call paralight.ca2604a9752bae31-cgu.15:? (paralight-b12df329bbbd1876+0xba5bff) (BuildId: 632355c6cdc6733a800d61fbdb55837a82c0d6c2)
    #8 __rust_try.llvm.2012152785340314153 paralight.ca2604a9752bae31-cgu.15:? (paralight-b12df329bbbd1876+0xbb6371) (BuildId: 632355c6cdc6733a800d61fbdb55837a82c0d6c2)
    #9 core::ops::function::FnOnce::call_once{{vtable.shim}} paralight.ca2604a9752bae31-cgu.15:? (paralight-b12df329bbbd1876+0xba7871) (BuildId: 632355c6cdc6733a800d61fbdb55837a82c0d6c2)
    #10 <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once std.143dd04dfe943471-cgu.09:? (paralight-b12df329bbbd1876+0xf0c788) (BuildId: 632355c6cdc6733a800d61fbdb55837a82c0d6c2)
    #11 std::sys::thread::unix::Thread::new::thread_start std.143dd04dfe943471-cgu.15:? (paralight-b12df329bbbd1876+0xf50e49) (BuildId: 632355c6cdc6733a800d61fbdb55837a82c0d6c2)

SUMMARY: ThreadSanitizer: data race ??:? in free
==================
test test_concurrent_thread_pools ... ok

test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.30s

ThreadSanitizer: reported 3 warnings
error: test failed, to rerun pass `--lib`

Note: Why would anyone want to spawn multiple Rayon pools and use them at the same time?

This is what happens when running a test suite with cargo test where each test case is creating its own thread pool (e.g. https://github.com/gendx/paralight/blob/main/src/lib.rs).
In a real-world program, there can be several components that independently spawn their own thread pool.

Oct 30 '25 09:10 gendx

Also reported as https://github.com/rayon-rs/rayon/issues/1275.

Oct 30 '25 09:10 gendx

This is likely a false positive since epoch uses fences which are not supported by TSan (see also the several existing closed issues). Consider using suppression file like: https://github.com/crossbeam-rs/crossbeam/blob/master/ci/tsan

Btw, does the same issue occur even when using https://github.com/crossbeam-rs/crossbeam/pull/998 (which improves compatibility with TSan in ways that may leads some false negatives)?

Oct 30 '25 09:10 taiki-e