Remotery icon indicating copy to clipboard operation
Remotery copied to clipboard

thread sanitizer warnings

Open conversy opened this issue 2 years ago • 11 comments

Hello,

with -fsanitize=thread, I get the following warning from remotery code. Looking at this discussion, it seems that volatileis not enough to guarantee a safe read in static rmtU32 LoadAcquire(rmtU32* volatile address). A mutex may be required... I'm not sure about which one to use so I cannot provide a quick fix, if ever it would be a fix.

Best,

Stéphane

WARNING: ThreadSanitizer: data race (pid=14734)
  Read of size 4 at 0x000108c05354 by thread T6 (mutexes: write M397):
    #0 LoadAcquire Remotery.c:585 (libdjnn-core.dylib:arm64+0x7ec6c)
    #1 rmtMessageQueue_AllocMessage Remotery.c:3999 (libdjnn-core.dylib:arm64+0x83dc0)
    #2 QueueThreadName Remotery.c:4894 (libdjnn-core.dylib:arm64+0x87cc4)
    #3 ThreadProfiler_Constructor Remotery.c:4937 (libdjnn-core.dylib:arm64+0x87bc8)
    #4 ThreadProfilers_GetThreadProfiler Remotery.c:5184 (libdjnn-core.dylib:arm64+0x87984)
    #5 ThreadProfilers_GetCurrentThreadProfiler Remotery.c:5210 (libdjnn-core.dylib:arm64+0x7f9e0)
    #6 _rmt_SetCurrentThreadName Remotery.c:6797 (libdjnn-core.dylib:arm64+0x7f928)
    #7 GatherThreadsLoop Remotery.c:5300 (libdjnn-core.dylib:arm64+0x83598)
    #8 StartFunc Remotery.c:1972 (libdjnn-core.dylib:arm64+0x842b4)

  Previous atomic write of size 4 at 0x000108c05354 by thread T5:
    #0 __tsan_atomic32_compare_exchange_val <null>:31683908 (libclang_rt.tsan_osx_dynamic.dylib:arm64e+0x577a4)
    #1 AtomicCompareAndSwap Remotery.c:521 (libdjnn-core.dylib:arm64+0x83f84)
    #2 rmtMessageQueue_AllocMessage Remotery.c:4008 (libdjnn-core.dylib:arm64+0x83e10)
    #3 QueueSampleTree Remotery.c:4796 (libdjnn-core.dylib:arm64+0x834a8)
    #4 ThreadProfiler_Pop Remotery.c:5000 (libdjnn-core.dylib:arm64+0x80404)
    #5 _rmt_EndCPUSample Remotery.c:6945 (libdjnn-core.dylib:arm64+0x80260)
    #6 Remotery_ThreadMain Remotery.c:6360 (libdjnn-core.dylib:arm64+0x81830)
    #7 StartFunc Remotery.c:1972 (libdjnn-core.dylib:arm64+0x842b4)

  Location is heap block of size 24 at 0x000108c05340 allocated by main thread:
    #0 malloc <null>:31683908 (libclang_rt.tsan_osx_dynamic.dylib:arm64e+0x592cc)
    #1 CRTMalloc Remotery.c:6603 (libdjnn-core.dylib:arm64+0x7edc0)
    #2 rmtMalloc Remotery.c:182 (libdjnn-core.dylib:arm64+0x7ef44)
    #3 Remotery_Constructor Remotery.c:6490 (libdjnn-core.dylib:arm64+0x7f0a4)
    #4 _rmt_CreateGlobalInstance Remotery.c:6657 (libdjnn-core.dylib:arm64+0x7ee8c)
    #5 djnn::init_core() core.cpp:74 (libdjnn-core.dylib:arm64+0x2cc4)
    #6 main <null>:31683908 (volta:arm64+0x100033ff4)

  Mutex M397 (0x000111c0a830) created at:
    #0 pthread_mutex_init <null>:31683908 (libclang_rt.tsan_osx_dynamic.dylib:arm64e+0x2d1f4)
    #1 mtxInit Remotery.c:474 (libdjnn-core.dylib:arm64+0x831c8)
    #2 ThreadProfilers_Constructor Remotery.c:5097 (libdjnn-core.dylib:arm64+0x81450)
    #3 Remotery_Constructor Remotery.c:6553 (libdjnn-core.dylib:arm64+0x7f3f8)
    #4 _rmt_CreateGlobalInstance Remotery.c:6657 (libdjnn-core.dylib:arm64+0x7ee8c)
    #5 djnn::init_core() core.cpp:74 (libdjnn-core.dylib:arm64+0x2cc4)
    #6 main <null>:31683908 (volta:arm64+0x100033ff4)

  Thread T6 (tid=53203, running) created by thread T4 at:
    #0 pthread_create <null>:31683908 (libclang_rt.tsan_osx_dynamic.dylib:arm64e+0x2bbe8)
    #1 rmtThread_Constructor Remotery.c:2013 (libdjnn-core.dylib:arm64+0x8169c)
    #2 InitThreadSampling Remotery.c:5576 (libdjnn-core.dylib:arm64+0x83324)
    #3 SampleThreadsLoop Remotery.c:5607 (libdjnn-core.dylib:arm64+0x8329c)
    #4 StartFunc Remotery.c:1972 (libdjnn-core.dylib:arm64+0x842b4)

  Thread T5 (tid=53202, running) created by main thread at:
    #0 pthread_create <null>:31683908 (libclang_rt.tsan_osx_dynamic.dylib:arm64e+0x2bbe8)
    #1 rmtThread_Constructor Remotery.c:2013 (libdjnn-core.dylib:arm64+0x8169c)
    #2 Remotery_Constructor Remotery.c:6565 (libdjnn-core.dylib:arm64+0x7f4b4)
    #3 _rmt_CreateGlobalInstance Remotery.c:6657 (libdjnn-core.dylib:arm64+0x7ee8c)
    #4 djnn::init_core() core.cpp:74 (libdjnn-core.dylib:arm64+0x2cc4)
    #5 main <null>:31683908 (volta:arm64+0x100033ff4)

conversy avatar May 07 '22 07:05 conversy

btw, to prevent another warning, I had to exchange some code, because ThreadProfilers_Constructor creates a thread:

    // Create the thread profilers container
    New_2(ThreadProfilers, rmt->threadProfilers, &rmt->timer, rmt->mq_to_rmt_thread);
    if (error != RMT_ERROR_NONE)
    {
        return error;
    }

    // Set as the global instance before creating any threads that uses it for sampling itself
    assert(g_Remotery == NULL);
    g_Remotery = rmt;
    g_RemoteryCreated = RMT_TRUE;

=>

    // Set as the global instance before creating any threads that uses it for sampling itself
    assert(g_Remotery == NULL);
    g_Remotery = rmt;
    g_RemoteryCreated = RMT_TRUE;

    // Create the thread profilers container
    New_2(ThreadProfilers, rmt->threadProfilers, &rmt->timer, rmt->mq_to_rmt_thread);
    if (error != RMT_ERROR_NONE)
    {
        return error;
    }

conversy avatar May 07 '22 11:05 conversy

Thank you.

There is scope in the future for use of std::atomic in C++ only code. It's unfortunate that the spec chooses to encode this in library primitives but we will likely have to deal with it.

Can you tell me what that last warning was? The comment is referring to Remotery's own use of rmt_BeginCPUSample to measure itself in the thread created by:

    // Create the main update thread once everything has been defined for the global remotery object
    New_2(rmtThread, rmt->thread, Remotery_ThreadMain, rmt);

dwilliamson avatar May 18 '22 20:05 dwilliamson

Hi, sorry I got caught by other concerns, I did not follow up this issue, I'm back to it.

I still have the problem with the latest Remoteryversion (though I did not use the code exchange this time). I got this now:

WARNING: ThreadSanitizer: data race (pid=77280)
  Read of size 4 at 0x00010a204594 by main thread (mutexes: write M322, write M317):
    #0 LoadAcquire Remotery.c:717 (libdjnn-core.dylib:arm64+0xc46fc)
    #1 rmtMessageQueue_AllocMessage Remotery.c:4207 (libdjnn-core.dylib:arm64+0xc9238)
    #2 QueueAddToStringTable Remotery.c:5035 (libdjnn-core.dylib:arm64+0xd9c20)
    #3 ThreadProfiler_Constructor Remotery.c:5132 (libdjnn-core.dylib:arm64+0xd9978)
    #4 ThreadProfilers_GetThreadProfiler Remotery.c:5420 (libdjnn-core.dylib:arm64+0xd9638)
    #5 ThreadProfilers_GetCurrentThreadProfiler Remotery.c:5446 (libdjnn-core.dylib:arm64+0xc6630)
    #6 _rmt_BeginCPUSample Remotery.c:7297 (libdjnn-core.dylib:arm64+0xc6c84)
    #7 djnn::GLSDLWindow::handle_event(SDL_Event&) gl_sdl_window.cpp:423 (libdjnn-gui.dylib:arm64+0x3b16fc)
    #8 djnn::SDLMainloop::handle_single_event(SDL_Event&) sdl_mainloop.cpp:262 (libdjnn-display.dylib:arm64+0x2b194)
    #9 djnn::SDLMainloop::handle_events(SDL_Event&) sdl_mainloop.cpp:201 (libdjnn-display.dylib:arm64+0x2abc4)
    #10 djnn::SDLMainloop::sdl_run() sdl_mainloop.cpp:159 (libdjnn-display.dylib:arm64+0x2a8f8)
    #11 djnn::SDLMainloop::run() sdl_mainloop.cpp:102 (libdjnn-display.dylib:arm64+0x2a800)
    #12 djnn::MainLoop::impl_activate() main_loop.cpp:102 (libdjnn-exec_env.dylib:arm64+0xc7e8)
    #13 djnn::CoreProcess::activate() process.cpp:309 (libdjnn-core.dylib:arm64+0x9a0c)
    #14 main main.cpp:36 (button_app:arm64+0x1000095fc)

  Previous atomic write of size 4 at 0x00010a204594 by thread T2:
    #0 __tsan_atomic32_compare_exchange_val <null>:74429764 (libclang_rt.tsan_osx_dynamic.dylib:arm64e+0x577a4)
    #1 AtomicCompareAndSwapU32 Remotery.c:633 (libdjnn-core.dylib:arm64+0xdb230)
    #2 rmtMessageQueue_AllocMessage Remotery.c:4216 (libdjnn-core.dylib:arm64+0xc92e0)
    #3 QueueSampleTree Remotery.c:5005 (libdjnn-core.dylib:arm64+0xd02c8)
    #4 ThreadProfiler_Pop Remotery.c:5227 (libdjnn-core.dylib:arm64+0xc75c0)
    #5 _rmt_EndCPUSample Remotery.c:7331 (libdjnn-core.dylib:arm64+0xc7264)
    #6 Remotery_ThreadMain Remotery.c:6717 (libdjnn-core.dylib:arm64+0xcbaf4)
    #7 StartFunc Remotery.c:2158 (libdjnn-core.dylib:arm64+0xd223c)

  Location is heap block of size 24 at 0x00010a204580 allocated by main thread:
    #0 malloc <null>:74429764 (libclang_rt.tsan_osx_dynamic.dylib:arm64e+0x592cc)
    #1 CRTMalloc Remotery.c:6991 (libdjnn-core.dylib:arm64+0xc49b4)
    #2 rmtMalloc Remotery.c:218 (libdjnn-core.dylib:arm64+0xc4c60)
    #3 Remotery_Constructor Remotery.c:6867 (libdjnn-core.dylib:arm64+0xc51f0)
    #4 _rmt_CreateGlobalInstance Remotery.c:7045 (libdjnn-core.dylib:arm64+0xc4b48)
    #5 djnn::init_core() core.cpp:74 (libdjnn-core.dylib:arm64+0x3540)
    #6 main main.cpp:37 (button_app:arm64+0x100007fa0)

  Mutex M322 (0x00010a102440) created at:
    #0 pthread_mutex_init <null>:74429764 (libclang_rt.tsan_osx_dynamic.dylib:arm64e+0x2d1f4)
    #1 SDL_CreateMutex_REAL <null>:74429764 (libSDL2-2.0.0.dylib:arm64+0xba8c4)
    #2 djnn::init_global_mutex() global_mutex.cpp:32 (libdjnn-exec_env.dylib:arm64+0xb9dc)
    #3 djnn::init_exec_env() exec_env.cpp:52 (libdjnn-exec_env.dylib:arm64+0x6f24)
    #4 main main.cpp:38 (button_app:arm64+0x100007fa4)

  Mutex M317 (0x00010c00b030) created at:
    #0 pthread_mutex_init <null>:74429764 (libclang_rt.tsan_osx_dynamic.dylib:arm64e+0x2d1f4)
    #1 mtxInit Remotery.c:576 (libdjnn-core.dylib:arm64+0xc99e0)
    #2 ThreadProfilers_Constructor Remotery.c:5342 (libdjnn-core.dylib:arm64+0xcb080)
    #3 Remotery_Constructor Remotery.c:6917 (libdjnn-core.dylib:arm64+0xc580c)
    #4 _rmt_CreateGlobalInstance Remotery.c:7045 (libdjnn-core.dylib:arm64+0xc4b48)
    #5 djnn::init_core() core.cpp:74 (libdjnn-core.dylib:arm64+0x3540)
    #6 main main.cpp:37 (button_app:arm64+0x100007fa0)

  Thread T2 (tid=905905, running) created by main thread at:
    #0 pthread_create <null>:74429764 (libclang_rt.tsan_osx_dynamic.dylib:arm64e+0x2bbe8)
    #1 rmtThread_Constructor Remotery.c:2199 (libdjnn-core.dylib:arm64+0xcb95c)
    #2 Remotery_Constructor Remotery.c:6931 (libdjnn-core.dylib:arm64+0xc5b68)
    #3 _rmt_CreateGlobalInstance Remotery.c:7045 (libdjnn-core.dylib:arm64+0xc4b48)
    #4 djnn::init_core() core.cpp:74 (libdjnn-core.dylib:arm64+0x3540)
    #5 main main.cpp:37 (button_app:arm64+0x100007fa0)

SUMMARY: ThreadSanitizer: data race Remotery.c:717 in LoadAcquire

It's on macOS 12.4 (21F79).

I'm not quite sure about the meaning of your question ('what that last warning was?')...

Best,

Stéphane

conversy avatar Sep 05 '22 12:09 conversy

This looks somewhat safe, although I'm struggling to find a Mac OS primitive in C to express this: if the load of write_pos is stale it just means allocations can fail when the queue isn't releasing quickly enough. There appear to be no situations where a stale write_pos can allow you to allocate from the queue when there isn't enough memory, causing an overflow.

I have not used TSAN before: does it require use of C11 atomics to inform what's going on?

dwilliamson avatar Sep 05 '22 21:09 dwilliamson

In LoadAcquire, can you try inserting a call to OSMemoryBarrier() after the call to CompilerReadFence()? This will require including <libkern/OSAtomic.h>.

This all does need a very thorough cleanup on all platforms but I am lacking the time.

dwilliamson avatar Sep 06 '22 10:09 dwilliamson

I tried, but it's not better. The thing is, TSAN complains about the last line of:

static rmtU32 LoadAcquire(rmtAtomicU32* address)
{
    rmtU32 value = *address;

since rmtAtomicU32 is actually typedef volatile rmtU32 rmtAtomicU32, TSAN considers it as an unprotected read, since it's not really an atomic var?

This would suggest to use an actual atomic var, such as _Atomic(rmtU32) available in C11? With the associated functions such as atomic_load?

conversy avatar Sep 06 '22 14:09 conversy

I guess TSAN doesn't understand pre-C11 code bases. It also doesn't appear to support explicit fences, so I'll have to bring in some C11 primitives. I mean, that's no loss as I can just #ifdef things a bit. In this case it sees non-atomic access racing with the atomic access as more harmful than it actually is.

This seems like a very similar Issue: https://github.com/microsoft/mimalloc/issues/130

There are actually no fence primitives introduced for load/acquire & store/release. This is unnecessary on x86/x64 but they need to be there for other platforms.

dwilliamson avatar Sep 06 '22 14:09 dwilliamson

nice find for the similar issue...

I wrote a tentative fix, I've sent a PR. With these fixes, I have no more warning at initialization or at run-time, and Remotery seems still operational. I'm not quite sure of the fix, but you may consider it.

I still have a problem when closing the app though, but this can be investigated later (see below).

WARNING: ThreadSanitizer: thread leak (pid=56817)
  Thread T1 (tid=1402950, finished) created by main thread at:
    #0 pthread_create <null>:113358148 (libclang_rt.tsan_osx_dynamic.dylib:arm64e+0x2bbe8)
    #1 rmtThread_Constructor Remotery.c:2217 (libdjnn-core.dylib:arm64+0xcb81c)
    #2 ThreadProfilers_Constructor Remotery.c:5380 (libdjnn-core.dylib:arm64+0xcb04c)
    #3 Remotery_Constructor Remotery.c:6934 (libdjnn-core.dylib:arm64+0xc56d0)
    #4 _rmt_CreateGlobalInstance Remotery.c:7062 (libdjnn-core.dylib:arm64+0xc4a0c)
    #5 djnn::init_core() core.cpp:74 (libdjnn-core.dylib:arm64+0x33fc)
    #6 main simplest.cpp:25 (simplest_app:arm64+0x1000036ec)

SUMMARY: ThreadSanitizer: thread leak Remotery.c:2217 in rmtThread_Constructor

conversy avatar Sep 07 '22 12:09 conversy

Sorry, I missed this thread. Moved my post upon request. Might be a lot of overlap with previous comments:

I wanted to create a small threaded test to run it through Clang's thread sanitizer, and it seems to produce a lot of output. I'm not sure how many are false positives, but I think it would be great if we could fix them somehow, since I hope more ppl will use the thread sanitizer (and other tools) on their projects in the future.

I have a branch draft here: https://github.com/Celtoys/Remotery/pull/239

I hope I have setup the test correctly. Currently only tested on macOS, but using pthreads, it should work on linux too.

Compile & run with:

clang -std=c89 -DRMT_ENABLED=1 -g -O0 -fsanitize=thread -Ilib sample/thread.c lib/Remotery.c -o build/test && ./build/test

Here's the output for the online readers:

Creating remotery instance
Spawning 4 threads
==================
WARNING: ThreadSanitizer: data race (pid=72475)
  Read of size 4 at 0x7b0800001174 by thread T3 (mutexes: write M22):
    #0 LoadAcquire Remotery.c:717 (test:x86_64+0x100002d29)
    #1 rmtMessageQueue_AllocMessage Remotery.c:4217 (test:x86_64+0x100007e06)
    #2 QueueAddToStringTable Remotery.c:5048 (test:x86_64+0x10001917f)
    #3 ThreadProfiler_Constructor Remotery.c:5145 (test:x86_64+0x100018e94)
    #4 ThreadProfilers_GetThreadProfiler Remotery.c:5433 (test:x86_64+0x100018b18)
    #5 ThreadProfilers_GetCurrentThreadProfiler Remotery.c:5459 (test:x86_64+0x10000514d)
    #6 _rmt_SetCurrentThreadName Remotery.c:7197 (test:x86_64+0x10000501c)
    #7 thread_start_proxy thread.c:33 (test:x86_64+0x1000020b2)

  Previous atomic write of size 4 at 0x7b0800001174 by thread T2:
    #0 __tsan_atomic32_compare_exchange_val <null>:3 (libclang_rt.tsan_osx_dynamic.dylib:x86_64h+0x51b60)
    #1 AtomicCompareAndSwapU32 Remotery.c:633 (test:x86_64+0x10001a98f)
    #2 rmtMessageQueue_AllocMessage Remotery.c:4226 (test:x86_64+0x100007e98)
    #3 QueueSampleTree Remotery.c:5018 (test:x86_64+0x10000eda0)
    #4 ThreadProfiler_Pop Remotery.c:5240 (test:x86_64+0x100006187)
    #5 _rmt_EndCPUSample Remotery.c:7342 (test:x86_64+0x100005dc2)
    #6 Remotery_ThreadMain Remotery.c:6728 (test:x86_64+0x10000a7c5)
    #7 StartFunc Remotery.c:2158 (test:x86_64+0x100010e52)

  Location is heap block of size 24 at 0x7b0800001160 allocated by main thread:
    #0 malloc <null>:3 (libclang_rt.tsan_osx_dynamic.dylib:x86_64h+0x532ec)
    #1 CRTMalloc Remotery.c:7002 (test:x86_64+0x100002f9f)
    #2 rmtMalloc Remotery.c:218 (test:x86_64+0x100003214)
    #3 Remotery_Constructor Remotery.c:6878 (test:x86_64+0x1000039a9)
    #4 _rmt_CreateGlobalInstance Remotery.c:7056 (test:x86_64+0x100003126)
    #5 main thread.c:133 (test:x86_64+0x1000021cf)

  Mutex M22 (0x7bb80000b030) created at:
    #0 pthread_mutex_init <null>:3 (libclang_rt.tsan_osx_dynamic.dylib:x86_64h+0x2cd05)
    #1 mtxInit Remotery.c:576 (test:x86_64+0x100008578)
    #2 ThreadProfilers_Constructor Remotery.c:5355 (test:x86_64+0x100009cf9)
    #3 Remotery_Constructor Remotery.c:6928 (test:x86_64+0x100004217)
    #4 _rmt_CreateGlobalInstance Remotery.c:7056 (test:x86_64+0x100003126)
    #5 main thread.c:133 (test:x86_64+0x1000021cf)

  Thread T3 (tid=17607725, running) created by main thread at:
    #0 pthread_create <null>:3 (libclang_rt.tsan_osx_dynamic.dylib:x86_64h+0x2bb6f)
    #1 thread_create thread.c:68 (test:x86_64+0x100001f9c)
    #2 main thread.c:145 (test:x86_64+0x10000226f)

  Thread T2 (tid=17607724, running) created by main thread at:
    #0 pthread_create <null>:3 (libclang_rt.tsan_osx_dynamic.dylib:x86_64h+0x2bb6f)
    #1 rmtThread_Constructor Remotery.c:2199 (test:x86_64+0x10000a650)
    #2 Remotery_Constructor Remotery.c:6942 (test:x86_64+0x100004675)
    #3 _rmt_CreateGlobalInstance Remotery.c:7056 (test:x86_64+0x100003126)
    #4 main thread.c:133 (test:x86_64+0x1000021cf)

SUMMARY: ThreadSanitizer: data race Remotery.c:717 in LoadAcquire
==================
==================
WARNING: ThreadSanitizer: data race (pid=72475)
  Read of size 4 at 0x7b0800001170 by thread T3 (mutexes: write M22):
    #0 LoadAcquire Remotery.c:717 (test:x86_64+0x100002d29)
    #1 rmtMessageQueue_AllocMessage Remotery.c:4218 (test:x86_64+0x100007e16)
    #2 QueueAddToStringTable Remotery.c:5048 (test:x86_64+0x10001917f)
    #3 ThreadProfiler_Constructor Remotery.c:5145 (test:x86_64+0x100018e94)
    #4 ThreadProfilers_GetThreadProfiler Remotery.c:5433 (test:x86_64+0x100018b18)
    #5 ThreadProfilers_GetCurrentThreadProfiler Remotery.c:5459 (test:x86_64+0x10000514d)
    #6 _rmt_SetCurrentThreadName Remotery.c:7197 (test:x86_64+0x10000501c)
    #7 thread_start_proxy thread.c:33 (test:x86_64+0x1000020b2)

  Previous write of size 4 at 0x7b0800001170 by thread T2:
    #0 StoreRelease Remotery.c:732 (test:x86_64+0x10000f63d)
    #1 rmtMessageQueue_ConsumeNextMessage Remotery.c:4297 (test:x86_64+0x100015bab)
    #2 Remotery_ConsumeMessageQueue Remotery.c:6629 (test:x86_64+0x10001156e)
    #3 Remotery_ThreadMain Remotery.c:6725 (test:x86_64+0x10000a7bb)
    #4 StartFunc Remotery.c:2158 (test:x86_64+0x100010e52)

  Location is heap block of size 24 at 0x7b0800001160 allocated by main thread:
    #0 malloc <null>:3 (libclang_rt.tsan_osx_dynamic.dylib:x86_64h+0x532ec)
    #1 CRTMalloc Remotery.c:7002 (test:x86_64+0x100002f9f)
    #2 rmtMalloc Remotery.c:218 (test:x86_64+0x100003214)
    #3 Remotery_Constructor Remotery.c:6878 (test:x86_64+0x1000039a9)
    #4 _rmt_CreateGlobalInstance Remotery.c:7056 (test:x86_64+0x100003126)
    #5 main thread.c:133 (test:x86_64+0x1000021cf)

  Mutex M22 (0x7bb80000b030) created at:
    #0 pthread_mutex_init <null>:3 (libclang_rt.tsan_osx_dynamic.dylib:x86_64h+0x2cd05)
    #1 mtxInit Remotery.c:576 (test:x86_64+0x100008578)
    #2 ThreadProfilers_Constructor Remotery.c:5355 (test:x86_64+0x100009cf9)
    #3 Remotery_Constructor Remotery.c:6928 (test:x86_64+0x100004217)
    #4 _rmt_CreateGlobalInstance Remotery.c:7056 (test:x86_64+0x100003126)
    #5 main thread.c:133 (test:x86_64+0x1000021cf)

  Thread T3 (tid=17607725, running) created by main thread at:
    #0 pthread_create <null>:3 (libclang_rt.tsan_osx_dynamic.dylib:x86_64h+0x2bb6f)
    #1 thread_create thread.c:68 (test:x86_64+0x100001f9c)
    #2 main thread.c:145 (test:x86_64+0x10000226f)

  Thread T2 (tid=17607724, running) created by main thread at:
    #0 pthread_create <null>:3 (libclang_rt.tsan_osx_dynamic.dylib:x86_64h+0x2bb6f)
    #1 rmtThread_Constructor Remotery.c:2199 (test:x86_64+0x10000a650)
    #2 Remotery_Constructor Remotery.c:6942 (test:x86_64+0x100004675)
    #3 _rmt_CreateGlobalInstance Remotery.c:7056 (test:x86_64+0x100003126)
    #4 main thread.c:133 (test:x86_64+0x1000021cf)

SUMMARY: ThreadSanitizer: data race Remotery.c:717 in LoadAcquire
==================
Entering thread!
Entering thread!
Entering thread!
Entering thread!
==================
WARNING: ThreadSanitizer: data race (pid=72475)
  Atomic write of size 8 at 0x7b0c00003058 by thread T2:
    #0 __tsan_atomic64_compare_exchange_val <null>:3 (libclang_rt.tsan_osx_dynamic.dylib:x86_64h+0x51c70)
    #1 AtomicCompareAndSwapPointer Remotery.c:660 (test:x86_64+0x100010432)
    #2 ObjectAllocator_Push Remotery.c:2335 (test:x86_64+0x100010948)
    #3 ObjectAllocator_FreeRange Remotery.c:2423 (test:x86_64+0x10001071d)
    #4 FreeSamples Remotery.c:4938 (test:x86_64+0x10000ecdc)
    #5 Remotery_SendSampleTreeMessage Remotery.c:6415 (test:x86_64+0x1000155ca)
    #6 Remotery_ConsumeMessageQueue Remotery.c:6612 (test:x86_64+0x100011501)
    #7 Remotery_ThreadMain Remotery.c:6725 (test:x86_64+0x10000a7bb)
    #8 StartFunc Remotery.c:2158 (test:x86_64+0x100010e52)

  Previous read of size 8 at 0x7b0c00003058 by thread T3:
    #0 ObjectAllocator_Pop Remotery.c:2350 (test:x86_64+0x1000102f1)
    #1 ObjectAllocator_Alloc Remotery.c:2377 (test:x86_64+0x10000fc22)
    #2 SampleTree_Push Remotery.c:4849 (test:x86_64+0x10001a18a)
    #3 ThreadProfiler_Push Remotery.c:5185 (test:x86_64+0x100005b3f)
    #4 _rmt_BeginCPUSample Remotery.c:7312 (test:x86_64+0x100005871)
    #5 recursiveFunction thread.c:100 (test:x86_64+0x100002471)
    #6 Run thread.c:114 (test:x86_64+0x100002394)
    #7 thread_start_proxy thread.c:35 (test:x86_64+0x1000020e9)

  Location is heap block of size 48 at 0x7b0c00003030 allocated by thread T3:
    #0 malloc <null>:3 (libclang_rt.tsan_osx_dynamic.dylib:x86_64h+0x532ec)
    #1 CRTMalloc Remotery.c:7002 (test:x86_64+0x100002f9f)
    #2 rmtMalloc Remotery.c:218 (test:x86_64+0x100003214)
    #3 SampleTree_Constructor Remotery.c:4769 (test:x86_64+0x10001935c)
    #4 ThreadProfiler_Constructor Remotery.c:5148 (test:x86_64+0x100018f61)
    #5 ThreadProfilers_GetThreadProfiler Remotery.c:5433 (test:x86_64+0x100018b18)
    #6 ThreadProfilers_GetCurrentThreadProfiler Remotery.c:5459 (test:x86_64+0x10000514d)
    #7 _rmt_SetCurrentThreadName Remotery.c:7197 (test:x86_64+0x10000501c)
    #8 thread_start_proxy thread.c:33 (test:x86_64+0x1000020b2)

  Thread T2 (tid=17607724, running) created by main thread at:
    #0 pthread_create <null>:3 (libclang_rt.tsan_osx_dynamic.dylib:x86_64h+0x2bb6f)
    #1 rmtThread_Constructor Remotery.c:2199 (test:x86_64+0x10000a650)
    #2 Remotery_Constructor Remotery.c:6942 (test:x86_64+0x100004675)
    #3 _rmt_CreateGlobalInstance Remotery.c:7056 (test:x86_64+0x100003126)
    #4 main thread.c:133 (test:x86_64+0x1000021cf)

  Thread T3 (tid=17607725, running) created by main thread at:
    #0 pthread_create <null>:3 (libclang_rt.tsan_osx_dynamic.dylib:x86_64h+0x2bb6f)
    #1 thread_create thread.c:68 (test:x86_64+0x100001f9c)
    #2 main thread.c:145 (test:x86_64+0x10000226f)

SUMMARY: ThreadSanitizer: data race Remotery.c:660 in AtomicCompareAndSwapPointer
==================
==================
WARNING: ThreadSanitizer: data race (pid=72475)
  Read of size 8 at 0x7b1c00003870 by thread T3:
    #0 ObjectAllocator_Pop Remotery.c:2355 (test:x86_64+0x100010322)
    #1 ObjectAllocator_Alloc Remotery.c:2377 (test:x86_64+0x10000fc22)
    #2 SampleTree_Push Remotery.c:4864 (test:x86_64+0x10001a31d)
    #3 ThreadProfiler_Push Remotery.c:5185 (test:x86_64+0x100005b3f)
    #4 _rmt_BeginCPUSample Remotery.c:7312 (test:x86_64+0x100005871)
    #5 Run thread.c:112 (test:x86_64+0x100002388)
    #6 thread_start_proxy thread.c:35 (test:x86_64+0x1000020e9)

  Previous write of size 8 at 0x7b1c00003870 by thread T2:
    #0 FlattenSamples Remotery.c:4911 (test:x86_64+0x100010559)
    #1 FreeSamples Remotery.c:4933 (test:x86_64+0x10000ec85)
    #2 Remotery_SendSampleTreeMessage Remotery.c:6415 (test:x86_64+0x1000155ca)
    #3 Remotery_ConsumeMessageQueue Remotery.c:6612 (test:x86_64+0x100011501)
    #4 Remotery_ThreadMain Remotery.c:6725 (test:x86_64+0x10000a7bb)
    #5 StartFunc Remotery.c:2158 (test:x86_64+0x100010e52)

  Location is heap block of size 112 at 0x7b1c00003870 allocated by thread T3:
    #0 malloc <null>:3 (libclang_rt.tsan_osx_dynamic.dylib:x86_64h+0x532ec)
    #1 CRTMalloc Remotery.c:7002 (test:x86_64+0x100002f9f)
    #2 rmtMalloc Remotery.c:218 (test:x86_64+0x100003214)
    #3 ObjectAllocator_Alloc Remotery.c:2385 (test:x86_64+0x10000fc71)
    #4 SampleTree_Push Remotery.c:4864 (test:x86_64+0x10001a31d)
    #5 ThreadProfiler_Push Remotery.c:5185 (test:x86_64+0x100005b3f)
    #6 _rmt_BeginCPUSample Remotery.c:7312 (test:x86_64+0x100005871)
    #7 Run thread.c:112 (test:x86_64+0x100002388)
    #8 thread_start_proxy thread.c:35 (test:x86_64+0x1000020e9)

  Thread T3 (tid=17607725, running) created by main thread at:
    #0 pthread_create <null>:3 (libclang_rt.tsan_osx_dynamic.dylib:x86_64h+0x2bb6f)
    #1 thread_create thread.c:68 (test:x86_64+0x100001f9c)
    #2 main thread.c:145 (test:x86_64+0x10000226f)

  Thread T2 (tid=17607724, running) created by main thread at:
    #0 pthread_create <null>:3 (libclang_rt.tsan_osx_dynamic.dylib:x86_64h+0x2bb6f)
    #1 rmtThread_Constructor Remotery.c:2199 (test:x86_64+0x10000a650)
    #2 Remotery_Constructor Remotery.c:6942 (test:x86_64+0x100004675)
    #3 _rmt_CreateGlobalInstance Remotery.c:7056 (test:x86_64+0x100003126)
    #4 main thread.c:133 (test:x86_64+0x1000021cf)

SUMMARY: ThreadSanitizer: data race Remotery.c:2355 in ObjectAllocator_Pop
==================
==================
WARNING: ThreadSanitizer: data race (pid=72475)
  Read of size 8 at 0x7b1c00010300 by thread T3:
    #0 ObjectAllocator_Pop Remotery.c:2355 (test:x86_64+0x100010322)
    #1 ObjectAllocator_Alloc Remotery.c:2377 (test:x86_64+0x10000fc22)
    #2 SampleTree_Push Remotery.c:4849 (test:x86_64+0x10001a18a)
    #3 ThreadProfiler_Push Remotery.c:5185 (test:x86_64+0x100005b3f)
    #4 _rmt_BeginCPUSample Remotery.c:7312 (test:x86_64+0x100005871)
    #5 recursiveFunction thread.c:100 (test:x86_64+0x100002471)
    #6 Run thread.c:114 (test:x86_64+0x100002394)
    #7 thread_start_proxy thread.c:35 (test:x86_64+0x1000020e9)

  Previous write of size 8 at 0x7b1c00010300 by thread T2:
    #0 FlattenSamples Remotery.c:4911 (test:x86_64+0x100010559)
    #1 FreeSamples Remotery.c:4933 (test:x86_64+0x10000ec85)
    #2 Remotery_SendSampleTreeMessage Remotery.c:6415 (test:x86_64+0x1000155ca)
    #3 Remotery_ConsumeMessageQueue Remotery.c:6612 (test:x86_64+0x100011501)
    #4 Remotery_ThreadMain Remotery.c:6725 (test:x86_64+0x10000a7bb)
    #5 StartFunc Remotery.c:2158 (test:x86_64+0x100010e52)

  Location is heap block of size 112 at 0x7b1c00010300 allocated by thread T3:
    #0 malloc <null>:3 (libclang_rt.tsan_osx_dynamic.dylib:x86_64h+0x532ec)
    #1 CRTMalloc Remotery.c:7002 (test:x86_64+0x100002f9f)
    #2 rmtMalloc Remotery.c:218 (test:x86_64+0x100003214)
    #3 ObjectAllocator_Alloc Remotery.c:2385 (test:x86_64+0x10000fc71)
    #4 SampleTree_Push Remotery.c:4864 (test:x86_64+0x10001a31d)
    #5 ThreadProfiler_Push Remotery.c:5185 (test:x86_64+0x100005b3f)
    #6 _rmt_BeginCPUSample Remotery.c:7312 (test:x86_64+0x100005871)
    #7 Run thread.c:112 (test:x86_64+0x100002388)
    #8 thread_start_proxy thread.c:35 (test:x86_64+0x1000020e9)

  Thread T3 (tid=17607725, running) created by main thread at:
    #0 pthread_create <null>:3 (libclang_rt.tsan_osx_dynamic.dylib:x86_64h+0x2bb6f)
    #1 thread_create thread.c:68 (test:x86_64+0x100001f9c)
    #2 main thread.c:145 (test:x86_64+0x10000226f)

  Thread T2 (tid=17607724, running) created by main thread at:
    #0 pthread_create <null>:3 (libclang_rt.tsan_osx_dynamic.dylib:x86_64h+0x2bb6f)
    #1 rmtThread_Constructor Remotery.c:2199 (test:x86_64+0x10000a650)
    #2 Remotery_Constructor Remotery.c:6942 (test:x86_64+0x100004675)
    #3 _rmt_CreateGlobalInstance Remotery.c:7056 (test:x86_64+0x100003126)
    #4 main thread.c:133 (test:x86_64+0x1000021cf)

SUMMARY: ThreadSanitizer: data race Remotery.c:2355 in ObjectAllocator_Pop
==================
==================
WARNING: ThreadSanitizer: data race (pid=72475)
  Read of size 8 at 0x7b1c0001d260 by thread T3:
    #0 ObjectAllocator_Pop Remotery.c:2355 (test:x86_64+0x100010322)
    #1 ObjectAllocator_Alloc Remotery.c:2377 (test:x86_64+0x10000fc22)
    #2 SampleTree_Push Remotery.c:4849 (test:x86_64+0x10001a18a)
    #3 ThreadProfiler_Push Remotery.c:5185 (test:x86_64+0x100005b3f)
    #4 _rmt_BeginCPUSample Remotery.c:7312 (test:x86_64+0x100005871)
    #5 aggregateFunction thread.c:95 (test:x86_64+0x1000024c4)
    #6 Run thread.c:113 (test:x86_64+0x10000238d)
    #7 thread_start_proxy thread.c:35 (test:x86_64+0x1000020e9)

  Previous write of size 8 at 0x7b1c0001d260 by thread T2:
    #0 FlattenSamples Remotery.c:4911 (test:x86_64+0x100010559)
    #1 FreeSamples Remotery.c:4933 (test:x86_64+0x10000ec85)
    #2 Remotery_SendSampleTreeMessage Remotery.c:6415 (test:x86_64+0x1000155ca)
    #3 Remotery_ConsumeMessageQueue Remotery.c:6612 (test:x86_64+0x100011501)
    #4 Remotery_ThreadMain Remotery.c:6725 (test:x86_64+0x10000a7bb)
    #5 StartFunc Remotery.c:2158 (test:x86_64+0x100010e52)

  Location is heap block of size 112 at 0x7b1c0001d260 allocated by thread T3:
    #0 malloc <null>:3 (libclang_rt.tsan_osx_dynamic.dylib:x86_64h+0x532ec)
    #1 CRTMalloc Remotery.c:7002 (test:x86_64+0x100002f9f)
    #2 rmtMalloc Remotery.c:218 (test:x86_64+0x100003214)
    #3 ObjectAllocator_Alloc Remotery.c:2385 (test:x86_64+0x10000fc71)
    #4 SampleTree_Push Remotery.c:4864 (test:x86_64+0x10001a31d)
    #5 ThreadProfiler_Push Remotery.c:5185 (test:x86_64+0x100005b3f)
    #6 _rmt_BeginCPUSample Remotery.c:7312 (test:x86_64+0x100005871)
    #7 Run thread.c:112 (test:x86_64+0x100002388)
    #8 thread_start_proxy thread.c:35 (test:x86_64+0x1000020e9)

  Thread T3 (tid=17607725, running) created by main thread at:
    #0 pthread_create <null>:3 (libclang_rt.tsan_osx_dynamic.dylib:x86_64h+0x2bb6f)
    #1 thread_create thread.c:68 (test:x86_64+0x100001f9c)
    #2 main thread.c:145 (test:x86_64+0x10000226f)

  Thread T2 (tid=17607724, running) created by main thread at:
    #0 pthread_create <null>:3 (libclang_rt.tsan_osx_dynamic.dylib:x86_64h+0x2bb6f)
    #1 rmtThread_Constructor Remotery.c:2199 (test:x86_64+0x10000a650)
    #2 Remotery_Constructor Remotery.c:6942 (test:x86_64+0x100004675)
    #3 _rmt_CreateGlobalInstance Remotery.c:7056 (test:x86_64+0x100003126)
    #4 main thread.c:133 (test:x86_64+0x1000021cf)

SUMMARY: ThreadSanitizer: data race Remotery.c:2355 in ObjectAllocator_Pop
==================
==================
WARNING: ThreadSanitizer: data race (pid=72475)
  Atomic write of size 8 at 0x7b0c00006058 by thread T2:
    #0 __tsan_atomic64_compare_exchange_val <null>:3 (libclang_rt.tsan_osx_dynamic.dylib:x86_64h+0x51c70)
    #1 AtomicCompareAndSwapPointer Remotery.c:660 (test:x86_64+0x100010432)
    #2 ObjectAllocator_Push Remotery.c:2335 (test:x86_64+0x100010948)
    #3 ObjectAllocator_FreeRange Remotery.c:2423 (test:x86_64+0x10001071d)
    #4 FreeSamples Remotery.c:4938 (test:x86_64+0x10000ecdc)
    #5 Remotery_SendSampleTreeMessage Remotery.c:6415 (test:x86_64+0x1000155ca)
    #6 Remotery_ConsumeMessageQueue Remotery.c:6612 (test:x86_64+0x100011501)
    #7 Remotery_ThreadMain Remotery.c:6725 (test:x86_64+0x10000a7bb)
    #8 StartFunc Remotery.c:2158 (test:x86_64+0x100010e52)

  Previous read of size 8 at 0x7b0c00006058 by thread T5:
    #0 ObjectAllocator_Pop Remotery.c:2350 (test:x86_64+0x1000102f1)
    #1 ObjectAllocator_Alloc Remotery.c:2377 (test:x86_64+0x10000fc22)
    #2 SampleTree_Push Remotery.c:4864 (test:x86_64+0x10001a31d)
    #3 ThreadProfiler_Push Remotery.c:5185 (test:x86_64+0x100005b3f)
    #4 _rmt_BeginCPUSample Remotery.c:7312 (test:x86_64+0x100005871)
    #5 Run thread.c:112 (test:x86_64+0x100002388)
    #6 thread_start_proxy thread.c:35 (test:x86_64+0x1000020e9)

  Location is heap block of size 48 at 0x7b0c00006030 allocated by thread T5:
    #0 malloc <null>:3 (libclang_rt.tsan_osx_dynamic.dylib:x86_64h+0x532ec)
    #1 CRTMalloc Remotery.c:7002 (test:x86_64+0x100002f9f)
    #2 rmtMalloc Remotery.c:218 (test:x86_64+0x100003214)
    #3 SampleTree_Constructor Remotery.c:4769 (test:x86_64+0x10001935c)
    #4 ThreadProfiler_Constructor Remotery.c:5148 (test:x86_64+0x100018f61)
    #5 ThreadProfilers_GetThreadProfiler Remotery.c:5433 (test:x86_64+0x100018b18)
    #6 ThreadProfilers_GetCurrentThreadProfiler Remotery.c:5459 (test:x86_64+0x10000514d)
    #7 _rmt_SetCurrentThreadName Remotery.c:7197 (test:x86_64+0x10000501c)
    #8 thread_start_proxy thread.c:33 (test:x86_64+0x1000020b2)

  Thread T2 (tid=17607724, running) created by main thread at:
    #0 pthread_create <null>:3 (libclang_rt.tsan_osx_dynamic.dylib:x86_64h+0x2bb6f)
    #1 rmtThread_Constructor Remotery.c:2199 (test:x86_64+0x10000a650)
    #2 Remotery_Constructor Remotery.c:6942 (test:x86_64+0x100004675)
    #3 _rmt_CreateGlobalInstance Remotery.c:7056 (test:x86_64+0x100003126)
    #4 main thread.c:133 (test:x86_64+0x1000021cf)

  Thread T5 (tid=17607766, running) created by main thread at:
    #0 pthread_create <null>:3 (libclang_rt.tsan_osx_dynamic.dylib:x86_64h+0x2bb6f)
    #1 thread_create thread.c:68 (test:x86_64+0x100001f9c)
    #2 main thread.c:145 (test:x86_64+0x10000226f)

SUMMARY: ThreadSanitizer: data race Remotery.c:660 in AtomicCompareAndSwapPointer
==================
==================
WARNING: ThreadSanitizer: data race (pid=72475)
  Write of size 8 at 0x0001092cbcb8 by thread T5:
    #0 QueueSampleTree Remotery.c:5030 (test:x86_64+0x10000ee49)
    #1 ThreadProfiler_Pop Remotery.c:5240 (test:x86_64+0x100006187)
    #2 _rmt_EndCPUSample Remotery.c:7342 (test:x86_64+0x100005dc2)
    #3 Run thread.c:115 (test:x86_64+0x100002399)
    #4 thread_start_proxy thread.c:35 (test:x86_64+0x1000020e9)

  Previous read of size 4 at 0x0001092cbcbc by thread T2:
    #0 rmtMessageQueue_ConsumeNextMessage Remotery.c:4292 (test:x86_64+0x100015b57)
    #1 Remotery_ConsumeMessageQueue Remotery.c:6629 (test:x86_64+0x10001156e)
    #2 Remotery_ThreadMain Remotery.c:6725 (test:x86_64+0x10000a7bb)
    #3 StartFunc Remotery.c:2158 (test:x86_64+0x100010e52)

  Thread T5 (tid=17607766, running) created by main thread at:
    #0 pthread_create <null>:3 (libclang_rt.tsan_osx_dynamic.dylib:x86_64h+0x2bb6f)
    #1 thread_create thread.c:68 (test:x86_64+0x100001f9c)
    #2 main thread.c:145 (test:x86_64+0x10000226f)

  Thread T2 (tid=17607724, running) created by main thread at:
    #0 pthread_create <null>:3 (libclang_rt.tsan_osx_dynamic.dylib:x86_64h+0x2bb6f)
    #1 rmtThread_Constructor Remotery.c:2199 (test:x86_64+0x10000a650)
    #2 Remotery_Constructor Remotery.c:6942 (test:x86_64+0x100004675)
    #3 _rmt_CreateGlobalInstance Remotery.c:7056 (test:x86_64+0x100003126)
    #4 main thread.c:133 (test:x86_64+0x1000021cf)

SUMMARY: ThreadSanitizer: data race Remotery.c:5030 in QueueSampleTree
==================
==================
WARNING: ThreadSanitizer: data race (pid=72475)
  Write of size 8 at 0x0001092cec50 by thread T5:
    #0 rmtMessageQueue_AllocMessage Remotery.c:4230 (test:x86_64+0x100007ee1)
    #1 QueueSampleTree Remotery.c:5018 (test:x86_64+0x10000eda0)
    #2 ThreadProfiler_Pop Remotery.c:5240 (test:x86_64+0x100006187)
    #3 _rmt_EndCPUSample Remotery.c:7342 (test:x86_64+0x100005dc2)
    #4 Run thread.c:115 (test:x86_64+0x100002399)
    #5 thread_start_proxy thread.c:35 (test:x86_64+0x1000020e9)

  Previous read of size 8 at 0x0001092cec50 by thread T2:
    #0 bin_SampleTree Remotery.c:6295 (test:x86_64+0x10001656d)
    #1 Remotery_SendSampleTreeMessage Remotery.c:6406 (test:x86_64+0x10001553e)
    #2 Remotery_ConsumeMessageQueue Remotery.c:6612 (test:x86_64+0x100011501)
    #3 Remotery_ThreadMain Remotery.c:6725 (test:x86_64+0x10000a7bb)
    #4 StartFunc Remotery.c:2158 (test:x86_64+0x100010e52)

  Thread T5 (tid=17607766, running) created by main thread at:
    #0 pthread_create <null>:3 (libclang_rt.tsan_osx_dynamic.dylib:x86_64h+0x2bb6f)
    #1 thread_create thread.c:68 (test:x86_64+0x100001f9c)
    #2 main thread.c:145 (test:x86_64+0x10000226f)

  Thread T2 (tid=17607724, running) created by main thread at:
    #0 pthread_create <null>:3 (libclang_rt.tsan_osx_dynamic.dylib:x86_64h+0x2bb6f)
    #1 rmtThread_Constructor Remotery.c:2199 (test:x86_64+0x10000a650)
    #2 Remotery_Constructor Remotery.c:6942 (test:x86_64+0x100004675)
    #3 _rmt_CreateGlobalInstance Remotery.c:7056 (test:x86_64+0x100003126)
    #4 main thread.c:133 (test:x86_64+0x1000021cf)

SUMMARY: ThreadSanitizer: data race Remotery.c:4230 in rmtMessageQueue_AllocMessage
==================
==================
WARNING: ThreadSanitizer: data race (pid=72475)
  Write of size 8 at 0x0001092cec90 by thread T4:
    #0 QueueSampleTree Remotery.c:5028 (test:x86_64+0x10000ede8)
    #1 ThreadProfiler_Pop Remotery.c:5240 (test:x86_64+0x100006187)
    #2 _rmt_EndCPUSample Remotery.c:7342 (test:x86_64+0x100005dc2)
    #3 Run thread.c:115 (test:x86_64+0x100002399)
    #4 thread_start_proxy thread.c:35 (test:x86_64+0x1000020e9)

  Previous read of size 4 at 0x0001092cec94 by thread T2:
    #0 bin_SampleTree Remotery.c:6324 (test:x86_64+0x10001681b)
    #1 Remotery_SendSampleTreeMessage Remotery.c:6406 (test:x86_64+0x10001553e)
    #2 Remotery_ConsumeMessageQueue Remotery.c:6612 (test:x86_64+0x100011501)
    #3 Remotery_ThreadMain Remotery.c:6725 (test:x86_64+0x10000a7bb)
    #4 StartFunc Remotery.c:2158 (test:x86_64+0x100010e52)

  Thread T4 (tid=17607726, running) created by main thread at:
    #0 pthread_create <null>:3 (libclang_rt.tsan_osx_dynamic.dylib:x86_64h+0x2bb6f)
    #1 thread_create thread.c:68 (test:x86_64+0x100001f9c)
    #2 main thread.c:145 (test:x86_64+0x10000226f)

  Thread T2 (tid=17607724, running) created by main thread at:
    #0 pthread_create <null>:3 (libclang_rt.tsan_osx_dynamic.dylib:x86_64h+0x2bb6f)
    #1 rmtThread_Constructor Remotery.c:2199 (test:x86_64+0x10000a650)
    #2 Remotery_Constructor Remotery.c:6942 (test:x86_64+0x100004675)
    #3 _rmt_CreateGlobalInstance Remotery.c:7056 (test:x86_64+0x100003126)
    #4 main thread.c:133 (test:x86_64+0x1000021cf)

SUMMARY: ThreadSanitizer: data race Remotery.c:5028 in QueueSampleTree
==================
==================
WARNING: ThreadSanitizer: data race (pid=72475)
  Write of size 8 at 0x0001092cbd28 by thread T6:
    #0 QueueSampleTree Remotery.c:5030 (test:x86_64+0x10000ee49)
    #1 ThreadProfiler_Pop Remotery.c:5240 (test:x86_64+0x100006187)
    #2 _rmt_EndCPUSample Remotery.c:7342 (test:x86_64+0x100005dc2)
    #3 Run thread.c:115 (test:x86_64+0x100002399)
    #4 thread_start_proxy thread.c:35 (test:x86_64+0x1000020e9)

  Previous read of size 4 at 0x0001092cbd28 by thread T2:
    #0 LoadAcquire Remotery.c:717 (test:x86_64+0x100002d29)
    #1 rmtMessageQueue_PeekNextMessage Remotery.c:4269 (test:x86_64+0x100002cb7)
    #2 Remotery_ConsumeMessageQueue Remotery.c:6591 (test:x86_64+0x100011441)
    #3 Remotery_ThreadMain Remotery.c:6725 (test:x86_64+0x10000a7bb)
    #4 StartFunc Remotery.c:2158 (test:x86_64+0x100010e52)

  Thread T6 (tid=17607767, running) created by main thread at:
    #0 pthread_create <null>:3 (libclang_rt.tsan_osx_dynamic.dylib:x86_64h+0x2bb6f)
    #1 thread_create thread.c:68 (test:x86_64+0x100001f9c)
    #2 main thread.c:145 (test:x86_64+0x10000226f)

  Thread T2 (tid=17607724, running) created by main thread at:
    #0 pthread_create <null>:3 (libclang_rt.tsan_osx_dynamic.dylib:x86_64h+0x2bb6f)
    #1 rmtThread_Constructor Remotery.c:2199 (test:x86_64+0x10000a650)
    #2 Remotery_Constructor Remotery.c:6942 (test:x86_64+0x100004675)
    #3 _rmt_CreateGlobalInstance Remotery.c:7056 (test:x86_64+0x100003126)
    #4 main thread.c:133 (test:x86_64+0x1000021cf)

SUMMARY: ThreadSanitizer: data race Remotery.c:5030 in QueueSampleTree
==================
==================
WARNING: ThreadSanitizer: data race (pid=72475)
  Write of size 4 at 0x0001092cbd7c by thread T3:
    #0 rmtMessageQueue_AllocMessage Remotery.c:4229 (test:x86_64+0x100007ebb)
    #1 QueueSampleTree Remotery.c:5018 (test:x86_64+0x10000eda0)
    #2 ThreadProfiler_Pop Remotery.c:5240 (test:x86_64+0x100006187)
    #3 _rmt_EndCPUSample Remotery.c:7342 (test:x86_64+0x100005dc2)
    #4 Run thread.c:115 (test:x86_64+0x100002399)
    #5 thread_start_proxy thread.c:35 (test:x86_64+0x1000020e9)

  Previous read of size 8 at 0x0001092cbd78 by thread T2:
    #0 Remotery_SendSampleTreeMessage Remotery.c:6415 (test:x86_64+0x1000155b9)
    #1 Remotery_ConsumeMessageQueue Remotery.c:6612 (test:x86_64+0x100011501)
    #2 Remotery_ThreadMain Remotery.c:6725 (test:x86_64+0x10000a7bb)
    #3 StartFunc Remotery.c:2158 (test:x86_64+0x100010e52)

  Thread T3 (tid=17607725, running) created by main thread at:
    #0 pthread_create <null>:3 (libclang_rt.tsan_osx_dynamic.dylib:x86_64h+0x2bb6f)
    #1 thread_create thread.c:68 (test:x86_64+0x100001f9c)
    #2 main thread.c:145 (test:x86_64+0x10000226f)

  Thread T2 (tid=17607724, running) created by main thread at:
    #0 pthread_create <null>:3 (libclang_rt.tsan_osx_dynamic.dylib:x86_64h+0x2bb6f)
    #1 rmtThread_Constructor Remotery.c:2199 (test:x86_64+0x10000a650)
    #2 Remotery_Constructor Remotery.c:6942 (test:x86_64+0x100004675)
    #3 _rmt_CreateGlobalInstance Remotery.c:7056 (test:x86_64+0x100003126)
    #4 main thread.c:133 (test:x86_64+0x1000021cf)

SUMMARY: ThreadSanitizer: data race Remotery.c:4229 in rmtMessageQueue_AllocMessage
==================
==================
WARNING: ThreadSanitizer: data race (pid=72475)
  Write of size 8 at 0x0001092cbd80 by thread T3:
    #0 rmtMessageQueue_AllocMessage Remotery.c:4230 (test:x86_64+0x100007ee1)
    #1 QueueSampleTree Remotery.c:5018 (test:x86_64+0x10000eda0)
    #2 ThreadProfiler_Pop Remotery.c:5240 (test:x86_64+0x100006187)
    #3 _rmt_EndCPUSample Remotery.c:7342 (test:x86_64+0x100005dc2)
    #4 Run thread.c:115 (test:x86_64+0x100002399)
    #5 thread_start_proxy thread.c:35 (test:x86_64+0x1000020e9)

  Previous read of size 8 at 0x0001092cbd80 by thread T2:
    [failed to restore the stack]

  Thread T3 (tid=17607725, running) created by main thread at:
    #0 pthread_create <null>:3 (libclang_rt.tsan_osx_dynamic.dylib:x86_64h+0x2bb6f)
    #1 thread_create thread.c:68 (test:x86_64+0x100001f9c)
    #2 main thread.c:145 (test:x86_64+0x10000226f)

  Thread T2 (tid=17607724, running) created by main thread at:
    #0 pthread_create <null>:3 (libclang_rt.tsan_osx_dynamic.dylib:x86_64h+0x2bb6f)
    #1 rmtThread_Constructor Remotery.c:2199 (test:x86_64+0x10000a650)
    #2 Remotery_Constructor Remotery.c:6942 (test:x86_64+0x100004675)
    #3 _rmt_CreateGlobalInstance Remotery.c:7056 (test:x86_64+0x100003126)
    #4 main thread.c:133 (test:x86_64+0x1000021cf)

SUMMARY: ThreadSanitizer: data race Remotery.c:4230 in rmtMessageQueue_AllocMessage
==================
==================
WARNING: ThreadSanitizer: data race (pid=72475)
  Write of size 8 at 0x0001092cbd88 by thread T3:
    #0 QueueSampleTree Remotery.c:5028 (test:x86_64+0x10000ede8)
    #1 ThreadProfiler_Pop Remotery.c:5240 (test:x86_64+0x100006187)
    #2 _rmt_EndCPUSample Remotery.c:7342 (test:x86_64+0x100005dc2)
    #3 Run thread.c:115 (test:x86_64+0x100002399)
    #4 thread_start_proxy thread.c:35 (test:x86_64+0x1000020e9)

  Previous read of size 4 at 0x0001092cbd8c by thread T2:
    [failed to restore the stack]

  Thread T3 (tid=17607725, running) created by main thread at:
    #0 pthread_create <null>:3 (libclang_rt.tsan_osx_dynamic.dylib:x86_64h+0x2bb6f)
    #1 thread_create thread.c:68 (test:x86_64+0x100001f9c)
    #2 main thread.c:145 (test:x86_64+0x10000226f)

  Thread T2 (tid=17607724, running) created by main thread at:
    #0 pthread_create <null>:3 (libclang_rt.tsan_osx_dynamic.dylib:x86_64h+0x2bb6f)
    #1 rmtThread_Constructor Remotery.c:2199 (test:x86_64+0x10000a650)
    #2 Remotery_Constructor Remotery.c:6942 (test:x86_64+0x100004675)
    #3 _rmt_CreateGlobalInstance Remotery.c:7056 (test:x86_64+0x100003126)
    #4 main thread.c:133 (test:x86_64+0x1000021cf)

SUMMARY: ThreadSanitizer: data race Remotery.c:5028 in QueueSampleTree
==================
==================
WARNING: ThreadSanitizer: data race (pid=72475)
  Write of size 8 at 0x0001092cbd98 by thread T3:
    #0 QueueSampleTree Remotery.c:5030 (test:x86_64+0x10000ee49)
    #1 ThreadProfiler_Pop Remotery.c:5240 (test:x86_64+0x100006187)
    #2 _rmt_EndCPUSample Remotery.c:7342 (test:x86_64+0x100005dc2)
    #3 Run thread.c:115 (test:x86_64+0x100002399)
    #4 thread_start_proxy thread.c:35 (test:x86_64+0x1000020e9)

  Previous read of size 4 at 0x0001092cbd98 by thread T2:
    [failed to restore the stack]

  Thread T3 (tid=17607725, running) created by main thread at:
    #0 pthread_create <null>:3 (libclang_rt.tsan_osx_dynamic.dylib:x86_64h+0x2bb6f)
    #1 thread_create thread.c:68 (test:x86_64+0x100001f9c)
    #2 main thread.c:145 (test:x86_64+0x10000226f)

  Thread T2 (tid=17607724, running) created by main thread at:
    #0 pthread_create <null>:3 (libclang_rt.tsan_osx_dynamic.dylib:x86_64h+0x2bb6f)
    #1 rmtThread_Constructor Remotery.c:2199 (test:x86_64+0x10000a650)
    #2 Remotery_Constructor Remotery.c:6942 (test:x86_64+0x100004675)
    #3 _rmt_CreateGlobalInstance Remotery.c:7056 (test:x86_64+0x100003126)
    #4 main thread.c:133 (test:x86_64+0x1000021cf)

SUMMARY: ThreadSanitizer: data race Remotery.c:5030 in QueueSampleTree

JCash avatar Sep 09 '22 17:09 JCash

thanks for investigating this, meanwhile I've done a PR, maybe you can have a look at it?

conversy avatar Sep 10 '22 07:09 conversy

@conversy Can you verify your tests on latest for this?

dwilliamson avatar Sep 14 '22 15:09 dwilliamson