folly icon indicating copy to clipboard operation
folly copied to clipboard

Coredump when running global_executor_test

Open dangleptr opened this issue 5 years ago • 14 comments

[heng@host02-cluster _build]$ ./global_executor_test
[==========] Running 2 tests from 1 test case.
[----------] Global test environment set-up.
[----------] 2 tests from GlobalExecutorTest
[ RUN      ] GlobalExecutorTest.GlobalCPUExecutor
[       OK ] GlobalExecutorTest.GlobalCPUExecutor (0 ms)
[ RUN      ] GlobalExecutorTest.GlobalIOExecutor
[       OK ] GlobalExecutorTest.GlobalIOExecutor (1 ms)
[----------] 2 tests from GlobalExecutorTest (1 ms total)

[----------] Global test environment tear-down
[==========] 2 tests from 1 test case ran. (2 ms total)
[  PASSED  ] 2 tests.
*** Aborted at 1572509134 (Unix time, try 'date -d @1572509134') ***
*** Signal 11 (SIGSEGV) (0xe0) received by PID 30858 (pthread TID 0x7f5b86094700) (linux TID 30860) (code: address not mapped to object), stack trace: ***
    @ 000000000086f5af _ZN5folly10symbolizer12_GLOBAL__N_118innerSignalHandlerEiP9siginfo_tPv
                       /home/heng/workspace/folly/folly/experimental/symbolizer/SignalHandler.cpp:431
    @ 000000000086f654 _ZN5folly10symbolizer12_GLOBAL__N_113signalHandlerEiP9siginfo_tPv
                       /home/heng/workspace/folly/folly/experimental/symbolizer/SignalHandler.cpp:446
    @ 00007f5b86e685ef (unknown)
    @ 00000000007a02e7 _ZNK5folly14ThreadLocalPtrINS_20SingletonThreadLocalISt10shared_ptrINS_14RequestContextEENS_6detail10DefaultTagENS5_11DefaultMakeIS4_EEvE7WrapperEvvE3getEv
                       /home/heng/workspace/folly/folly/ThreadLocal.h:159
                       -> /home/heng/workspace/folly/folly/io/async/Request.cpp
    @ 000000000079a797 _ZNK5folly11ThreadLocalINS_20SingletonThreadLocalISt10shared_ptrINS_14RequestContextEENS_6detail10DefaultTagENS5_11DefaultMakeIS4_EEvE7WrapperEvvEdeEv
                       /home/heng/workspace/folly/folly/ThreadLocal.h:68
                       -> /home/heng/workspace/folly/folly/io/async/Request.cpp
    @ 00000000007980c0 _ZN5folly20SingletonThreadLocalISt10shared_ptrINS_14RequestContextEENS_6detail10DefaultTagENS4_11DefaultMakeIS3_EEvE10getWrapperEv
                       /home/heng/workspace/folly/folly/SingletonThreadLocal.h:148
                       -> /home/heng/workspace/folly/folly/io/async/Request.cpp
    @ 00000000007980d4 _ZN5folly20SingletonThreadLocalISt10shared_ptrINS_14RequestContextEENS_6detail10DefaultTagENS4_11DefaultMakeIS3_EEvE13LocalLifetimeD1Ev
                       /home/heng/workspace/folly/folly/SingletonThreadLocal.h:120
                       -> /home/heng/workspace/folly/folly/io/async/Request.cpp
    @ 00007f5b8761eeb5 _ZN12_GLOBAL__N_13runEPv
                       /home/heng/workspace/gcc-8.3.0/x86_64-pc-linux-gnu/libstdc++-v3/libsupc++/../../.././libstdc++-v3/libsupc++/atexit_thread.cc:75
    @ 00007f5b86e60c61 __nptl_deallocate_tsd
    @ 00007f5b86e60e72 start_thread
    @ 00007f5b86b8988c clone
Segmentation fault (core dumped)

version: v2019.10.14.00

dangleptr avatar Oct 31 '19 08:10 dangleptr

(gdb) bt
#0  0x000000000079cf5b in folly::ThreadLocalPtr<folly::SingletonThreadLocal<std::shared_ptr<folly::RequestContext>, folly::detail::DefaultTag, folly::detail::DefaultMake<std::shared_ptr<folly::RequestContext> >, void>::Wrapper, void, void>::get (this=0x7f4ad8001760) at /home/heng/workspace/folly/folly/ThreadLocal.h:159
#1  0x000000000079740c in folly::ThreadLocal<folly::SingletonThreadLocal<std::shared_ptr<folly::RequestContext>, folly::detail::DefaultTag, folly::detail::DefaultMake<std::shared_ptr<folly::RequestContext> >, void>::Wrapper, void, void>::get (this=0x7f4ad8001760) at /home/heng/workspace/folly/folly/ThreadLocal.h:68
#2  folly::ThreadLocal<folly::SingletonThreadLocal<std::shared_ptr<folly::RequestContext>, folly::detail::DefaultTag, folly::detail::DefaultMake<std::shared_ptr<folly::RequestContext> >, void>::Wrapper, void, void>::operator* (this=0x7f4ad8001760) at /home/heng/workspace/folly/folly/ThreadLocal.h:77
#3  0x0000000000794d35 in folly::SingletonThreadLocal<std::shared_ptr<folly::RequestContext>, folly::detail::DefaultTag, folly::detail::DefaultMake<std::shared_ptr<folly::RequestContext> >, void>::getWrapper
    () at /home/heng/workspace/folly/folly/SingletonThreadLocal.h:148
#4  0x0000000000794d49 in folly::SingletonThreadLocal<std::shared_ptr<folly::RequestContext>, folly::detail::DefaultTag, folly::detail::DefaultMake<std::shared_ptr<folly::RequestContext> >, void>::LocalLifetime::~LocalLifetime (this=0x7f4add6def78, __in_chrg=<optimized out>) at /home/heng/workspace/folly/folly/SingletonThreadLocal.h:120
#5  0x00007f4adec70eb6 in (anonymous namespace)::run (p=<optimized out>) at ../../.././libstdc++-v3/libsupc++/atexit_thread.cc:75
#6  0x00007f4ade4b2c62 in __nptl_deallocate_tsd () from /lib64/libpthread.so.0
#7  0x00007f4ade4b2e73 in start_thread () from /lib64/libpthread.so.0
#8  0x00007f4ade1db88d in clone () from /lib64/libc.so.6

dangleptr avatar Nov 20 '19 04:11 dangleptr

It seems that in IOThreadPool, the thread local variable has been released when accessing it. Any thoughts?

dangleptr avatar Nov 21 '19 05:11 dangleptr

@dangleptr What OS are you running? I run into this similar crash with CentOS74 + devtoolset7. So, I tried the same with a Ubuntu 19.10 Docker and don't see the same crash.

boringuy avatar Dec 23 '19 15:12 boringuy

centos 7.5 + gcc 8.3.0 @boringuy

dangleptr avatar Dec 24 '19 11:12 dangleptr

Anecdotally, observing this in mcrouter as well, which depends on folly: also on CentOS 7.5, GCC 5.x.

tobz avatar Jan 09 '20 19:01 tobz

I have not tried all the versions in between but confirmed the unit test cored with v2019.11.11.00 but does not core in v2019.12.30.00 and latest v2020.01.13.00

boringuy avatar Jan 16 '20 16:01 boringuy

Woohoo! I can confirm that v2019.12.30.00 also fixes my issue. I too did not try bisecting, but I can say that v2019.12.02.00 still segfaulted for me.

tobz avatar Jan 17 '20 22:01 tobz

thread_pool_executor_test SegFault on v2020.4.20 with the similar error.

#0  0x00000000009cac61 in folly::ThreadLocalPtr<folly::SingletonThreadLocal<folly::Optional<folly::BlockingContext>, folly::detail::DefaultTag, folly::detail::DefaultMake<folly::Optional<folly::BlockingContext> >, void>::Wrapper, void, void>::get (this=0x7f6e94000af0) at /home/vesoft/data/chenheng/3rd/folly/folly/ThreadLocal.h:160
#1  0x00000000009ca0dc in folly::ThreadLocal<folly::SingletonThreadLocal<folly::Optional<folly::BlockingContext>, folly::detail::DefaultTag, folly::detail::DefaultMake<folly::Optional<folly::BlockingContext> >, void>::Wrapper, void, void>::get (this=0x7f6e94000af0) at /home/vesoft/data/chenheng/3rd/folly/folly/ThreadLocal.h:69
#2  folly::ThreadLocal<folly::SingletonThreadLocal<folly::Optional<folly::BlockingContext>, folly::detail::DefaultTag, folly::detail::DefaultMake<folly::Optional<folly::BlockingContext> >, void>::Wrapper, void, void>::operator* (
    this=0x7f6e94000af0) at /home/vesoft/data/chenheng/3rd/folly/folly/ThreadLocal.h:78
#3  0x00000000009c9e89 in folly::SingletonThreadLocal<folly::Optional<folly::BlockingContext>, folly::detail::DefaultTag, folly::detail::DefaultMake<folly::Optional<folly::BlockingContext> >, void>::getWrapper ()
    at /home/vesoft/data/chenheng/3rd/folly/folly/SingletonThreadLocal.h:149
#4  0x00000000009c9e9d in folly::SingletonThreadLocal<folly::Optional<folly::BlockingContext>, folly::detail::DefaultTag, folly::detail::DefaultMake<folly::Optional<folly::BlockingContext> >, void>::LocalLifetime::~LocalLifetime (
    this=0x7f6e8effd508, __in_chrg=<optimized out>) at /home/vesoft/data/chenheng/3rd/folly/folly/SingletonThreadLocal.h:121
#5  0x00007f6ea01773b6 in (anonymous namespace)::run (p=<optimized out>) at ../../.././libstdc++-v3/libsupc++/atexit_thread.cc:75
#6  0x00007f6ea0474c62 in __nptl_deallocate_tsd () from /lib64/libpthread.so.0
#7  0x00007f6ea0474e73 in start_thread () from /lib64/libpthread.so.0
#8  0x00007f6e9f90188d in clone () from /lib64/libc.so.6

ThreadPoolExecutorTest.EDFBasic

dangleptr avatar Apr 27 '20 10:04 dangleptr

folly-v2023.02.20.00 unit test ./synchronized_test crashed, compiled by devtoolset-8 gcc 8.3 and centos 7.8 and kernel 3.10.0-957.21.3.el7

#0  0x00000000005c3a99 in folly::ThreadLocalPtr<folly::SingletonThreadLocal<folly::ThreadLocalPRNG::operator()()::Wrapper, folly::(anonymous namespace)::RandomTag>::Wrapper, folly::(anonymous namespace)::RandomTag, void>::get (this=0x7fffa4000930)
    at ../folly/detail/ThreadLocalDetail.h:322
#1  folly::ThreadLocal<folly::SingletonThreadLocal<folly::ThreadLocalPRNG::operator()()::Wrapper, folly::(anonymous namespace)::RandomTag>::Wrapper, folly::(anonymous namespace)::RandomTag, void>::get (this=0x7fffa4000930)
    at ../folly/ThreadLocal.h:69
#2  folly::ThreadLocal<folly::SingletonThreadLocal<folly::ThreadLocalPRNG::operator()()::Wrapper, folly::(anonymous namespace)::RandomTag>::Wrapper, folly::(anonymous namespace)::RandomTag, void>::operator* (this=0x7fffa4000930)
    at ../folly/ThreadLocal.h:78
#3  folly::SingletonThreadLocal<folly::ThreadLocalPRNG::operator()()::Wrapper, folly::(anonymous namespace)::RandomTag, folly::detail::DefaultMake<folly::ThreadLocalPRNG::operator()()::Wrapper>, folly::(anonymous namespace)::RandomTag>::getWrapper(void) () at ../folly/SingletonThreadLocal.h:138
#4  0x00000000005c3af9 in folly::SingletonThreadLocal<folly::ThreadLocalPRNG::operator()()::Wrapper, folly::(anonymous namespace)::RandomTag, folly::detail::DefaultMake<folly::ThreadLocalPRNG::operator()()::Wrapper>, folly::(anonymous namespace)::RandomTag>::LocalLifetime::~LocalLifetime(void) (this=0x7fffc765f4e0, __in_chrg=<optimized out>)
    at ../folly/SingletonThreadLocal.h:128
#5  0x00000000006cc626 in (anonymous namespace)::run(void*) ()
#6  0x00007ffff74aaca2 in __nptl_deallocate_tsd () from /usr/lib64/libpthread.so.0
#7  0x00007ffff74aaeb3 in start_thread () from /usr/lib64/libpthread.so.0
#8  0x00007ffff71d39fd in clone () from /usr/lib64/libc.so.6

lrita avatar Feb 24 '23 11:02 lrita

folly-v2023.02.20.00 unit test ./synchronized_test crashed, compiled by devtoolset-8 gcc 8.3 and centos 7.8 and kernel 3.10.0-957.21.3.el7

#0  0x00000000005c3a99 in folly::ThreadLocalPtr<folly::SingletonThreadLocal<folly::ThreadLocalPRNG::operator()()::Wrapper, folly::(anonymous namespace)::RandomTag>::Wrapper, folly::(anonymous namespace)::RandomTag, void>::get (this=0x7fffa4000930)
    at ../folly/detail/ThreadLocalDetail.h:322
#1  folly::ThreadLocal<folly::SingletonThreadLocal<folly::ThreadLocalPRNG::operator()()::Wrapper, folly::(anonymous namespace)::RandomTag>::Wrapper, folly::(anonymous namespace)::RandomTag, void>::get (this=0x7fffa4000930)
    at ../folly/ThreadLocal.h:69
#2  folly::ThreadLocal<folly::SingletonThreadLocal<folly::ThreadLocalPRNG::operator()()::Wrapper, folly::(anonymous namespace)::RandomTag>::Wrapper, folly::(anonymous namespace)::RandomTag, void>::operator* (this=0x7fffa4000930)
    at ../folly/ThreadLocal.h:78
#3  folly::SingletonThreadLocal<folly::ThreadLocalPRNG::operator()()::Wrapper, folly::(anonymous namespace)::RandomTag, folly::detail::DefaultMake<folly::ThreadLocalPRNG::operator()()::Wrapper>, folly::(anonymous namespace)::RandomTag>::getWrapper(void) () at ../folly/SingletonThreadLocal.h:138
#4  0x00000000005c3af9 in folly::SingletonThreadLocal<folly::ThreadLocalPRNG::operator()()::Wrapper, folly::(anonymous namespace)::RandomTag, folly::detail::DefaultMake<folly::ThreadLocalPRNG::operator()()::Wrapper>, folly::(anonymous namespace)::RandomTag>::LocalLifetime::~LocalLifetime(void) (this=0x7fffc765f4e0, __in_chrg=<optimized out>)
    at ../folly/SingletonThreadLocal.h:128
#5  0x00000000006cc626 in (anonymous namespace)::run(void*) ()
#6  0x00007ffff74aaca2 in __nptl_deallocate_tsd () from /usr/lib64/libpthread.so.0
#7  0x00007ffff74aaeb3 in start_thread () from /usr/lib64/libpthread.so.0
#8  0x00007ffff71d39fd in clone () from /usr/lib64/libc.so.6

CentOS 7.4 using glibc2.17,which has no __cxa_thread_atexit and __cxa_thread_atexit_impl support,then libstdc++ using TSD(pthread_key_create) implements thread_local variables' dtor invoking(which is shown as (anonymous namespace)::run(void*) () in the backtrace).

Unfortunately glibc TSD delete callbacks are not invoked in reverse order in which they were added. So this make ~LocalLifetime is invoked later than StaticMetaBase::onThreadExit.

lrita avatar Mar 06 '23 09:03 lrita

Seems this bug still exists. I ran into the same coredump when using the folly-2024.04.01. My environment is AliOS-7 (based on centos I believe) with libc 2.32 :

$ uname -a
Linux 2a24bff1ef3d 4.9.168-019.ali3000.alios7.x86_64 #1 SMP Thu Dec 31 20:20:01 CST 2020 x86_64 x86_64 x86_64 GNU/Linux
$ /lib64/libc.so.6
GNU C Library (GNU libc) release release version 2.32.
Copyright (C) 2020 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.
There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A
PARTICULAR PURPOSE.
Compiled by GNU CC version 10.2.1 20200825 (Alibaba 10.2.1-3 2.17).
libc ABIs: UNIQUE IFUNC ABSOLUTE
For bug reporting instructions, please see:
<https://www.gnu.org/software/libc/bugs.html>.

And the stacktrace is

*** Signal 11 (SIGSEGV) (0x7f59dde2a6b9) received by PID 3634169 (pthread TID 0x7f5e2cf06640) (linux TID 3634170) (code: address not mapped to object), stack trace: ***
    @ 000000000058190c folly::symbolizer::(anonymous namespace)::signalHandler(int, siginfo_t*, void*)
                       /code/thirdparty/src/folly-2024.04.01.00/folly/experimental/symbolizer/SignalHandler.cpp:449
    @ 00000000000140ff (unknown)
    @ 000000000050b96f folly::SingletonThreadLocal<folly::RequestContext::StaticContext, folly::RequestContext, folly::detail::DefaultMake<folly::RequestContext::StaticContext>, folly::RequestContext>::getWrapper()
                       /code/thirdparty/src/folly-2024.04.01.00/folly/ThreadLocal.h:151
                       -> /code/thirdparty/src/folly-2024.04.01.00/folly/io/async/Request.cpp
    @ 000000000050bac8 folly::SingletonThreadLocal<folly::RequestContext::StaticContext, folly::RequestContext, folly::detail::DefaultMake<folly::RequestContext::StaticContext>, folly::RequestContext>::LocalLifetime::~LocalLifetime()
                       /code/thirdparty/src/folly-2024.04.01.00/folly/SingletonThreadLocal.h:127
                       -> /code/thirdparty/src/folly-2024.04.01.00/folly/io/async/Request.cpp
    @ 00000000000a4a35 (unknown)
    @ 0000000000009210 __nptl_deallocate_tsd
    @ 000000000000940b start_thread
    @ 0000000000100302 clone

@lrita do you have any idea how to fix this? Thanks a lot.

HolyLow avatar May 27 '24 03:05 HolyLow

Following the instructions in README.md, I tried to compile and run all the test cases of folly on my dev machine, and it appears that a large amount of testcases failed because of segfault.

The following tests FAILED:
        336 - atomic_shared_ptr_test.AtomicSharedPtr.DeterministicTest (SEGFAULT
)
        341 - cache_locality_test.CacheLocality.LinuxActual (Failed)
        354 - core_cached_shared_ptr_test.CoreCachedSharedPtr.AtomicCoreCachedSh
aredPtr (SEGFAULT)
        403 - dynamic_bounded_queue_test.DynamicBoundedQueue.enqDeq (SEGFAULT)
        420 - unbounded_queue_test.UnboundedQueue.enqDeq (SEGFAULT)
        474 - executor_test.ManualExecutor.getViaDoesNotDeadlock (SEGFAULT)
        484 - fiber_io_executor_test.FiberIOExecutorTest.event_base (SEGFAULT)
        487 - global_executor_test.GlobalExecutorTest.GlobalImmutableIOExecutor
(SEGFAULT)
        489 - global_executor_test.GlobalExecutorTest.GlobalIOExecutor (SEGFAULT
)
        503 - threaded_executor_test.ThreadedExecutorTest.exception (SEGFAULT)
        516 - priority_unbounded_blocking_queue_test.PriorityUnboundedBlockingQu
eueTest.concurrent_push_pop (SEGFAULT)
        522 - unbounded_blocking_queue_test.UnboundedBlockingQueue.concurrentPus
hPop (SEGFAULT)
        581 - lock_free_ring_buffer_test.LockFreeRingBuffer.writesNeverFail (SEG
FAULT)
        775 - callback_lifetime_test.CallbackLifetimeTest.thenReturnsValue (SEGF
AULT)
        776 - callback_lifetime_test.CallbackLifetimeTest.thenReturnsValueThrows
 (SEGFAULT)
        777 - callback_lifetime_test.CallbackLifetimeTest.thenReturnsFuture (SEG
FAULT)
        778 - callback_lifetime_test.CallbackLifetimeTest.thenReturnsFutureThrow
s (SEGFAULT)
        779 - callback_lifetime_test.CallbackLifetimeTest.thenErrorTakesExnRetur
nsValueMatch (SEGFAULT)
        780 - callback_lifetime_test.CallbackLifetimeTest.thenErrorTakesExnRetur
nsValueMatchThrows (SEGFAULT)
        781 - callback_lifetime_test.CallbackLifetimeTest.thenErrorTakesExnRetur
nsValueWrong (SEGFAULT)
        782 - callback_lifetime_test.CallbackLifetimeTest.thenErrorTakesExnRetur
nsValueWrongThrows (SEGFAULT)
        783 - callback_lifetime_test.CallbackLifetimeTest.thenErrorTakesExnRetur
nsFutureMatch (SEGFAULT)
        784 - callback_lifetime_test.CallbackLifetimeTest.thenErrorTakesExnRetur
nsFutureMatchThrows (SEGFAULT)
        785 - callback_lifetime_test.CallbackLifetimeTest.thenErrorTakesExnRetur
nsFutureWrong (SEGFAULT)
        786 - callback_lifetime_test.CallbackLifetimeTest.thenErrorTakesExnRetur
nsFutureWrongThrows (SEGFAULT)
        787 - callback_lifetime_test.CallbackLifetimeTest.thenErrorTakesWrapRetu
rnsValue (SEGFAULT)
        788 - callback_lifetime_test.CallbackLifetimeTest.thenErrorTakesWrapRetu
rnsValueThrows (SEGFAULT)
        789 - callback_lifetime_test.CallbackLifetimeTest.thenErrorTakesWrapRetu
rnsFuture (SEGFAULT)
        790 - callback_lifetime_test.CallbackLifetimeTest.thenErrorTakesWrapRetu
rnsFutureThrows (SEGFAULT)
        904 - interrupt_test.Interrupt.futureWithinTimedOut (SEGFAULT)
        905 - interrupt_test.Interrupt.semiFutureWithinTimedOut (SEGFAULT)
        967 - retrying_test.RetryingTest.policyCappedJitteredExponentialBackoff
(SEGFAULT)
        968 - retrying_test.RetryingTest.policyCappedJitteredExponentialBackoffU
nsafe (SEGFAULT)
        971 - retrying_test.RetryingTest.policySleepDefaults (SEGFAULT)
        972 - retrying_test.RetryingTest.largeRetries (SEGFAULT)
        1031 - via_test.ViaFixture.threadHops (SEGFAULT)
        1032 - via_test.ViaFixture.chainVias (SEGFAULT)
        1034 - via_test.ViaFixture.viaAssignment (SEGFAULT)
        1038 - via_test.Via.viaThenGetWasRacy (SEGFAULT)
        1045 - via_test.Via.viaRaces (SEGFAULT)
        1063 - wait_test.Wait.waitWithDuration (SEGFAULT)
        1064 - wait_test.Wait.multipleWait (SEGFAULT)
        1065 - wait_test.Wait.WaitPlusThen (SEGFAULT)
        1066 - wait_test.Wait.cancelAfterWait (SEGFAULT)
        1080 - window_test.Window.parallel (SEGFAULT)
        1081 - window_test.Window.parallelWithError (SEGFAULT)
        1082 - window_test.Window.allParallelWithError (SEGFAULT)
        1283 - AsyncUDPSocketTest.AsyncSocketIntegrationTest.PingPong (SEGFAULT)
        1284 - AsyncUDPSocketTest.AsyncSocketIntegrationTest.PingPongNotify (SEG
FAULT)
        1285 - AsyncUDPSocketTest.AsyncSocketIntegrationTest.PingPongNotifyMmsg
(SEGFAULT)
        1286 - AsyncUDPSocketTest.AsyncSocketIntegrationTest.PingPongRecvTosDisa
bled (SEGFAULT)
        1287 - AsyncUDPSocketTest.AsyncSocketIntegrationTest.PingPongRecvTos (SE
GFAULT)
        1288 - AsyncUDPSocketTest.*/ConnectedAsyncSocketIntegrationTest.Connecte
dPingPong/* (SEGFAULT)
        1289 - AsyncUDPSocketTest.AsyncSocketIntegrationTest.PingPongPauseResume
Listening (SEGFAULT)
        1292 - AsyncUDPSocketTest.AsyncUDPSocketTest.TestErrToNonExistentServer
(SEGFAULT)
        1293 - AsyncUDPSocketTest.AsyncUDPSocketTest.TestUnsetErrCallback (SEGFA
ULT)
        1294 - AsyncUDPSocketTest.AsyncUDPSocketTest.CloseInErrorCallback (Timeo
ut)
        1295 - AsyncUDPSocketTest.AsyncUDPSocketTest.TestNonExistentServerNoErrC
b (SEGFAULT)
        1300 - AsyncUDPSocketTest.AsyncUDPSocketTest.TestDetachAttach (SEGFAULT)
        1301 - AsyncUDPSocketTest.AsyncUDPSocketTest.TestWriteCmsg (SEGFAULT)
        1302 - AsyncUDPSocketTest.AsyncUDPSocketTest.TestWriteDynamicCmsg (SEGFA
ULT)
        1304 - AsyncUDPSocketTest.AsyncUDPSocketTest.TestWritemCmsg (SEGFAULT)
        1305 - AsyncUDPSocketTest.AsyncUDPSocketTest.TestWritemDynamicCmsg (SEGF
AULT)
        1307 - AsyncUDPSocketTest.AsyncUDPSocketTest.TestWritemNontrivialCmsgs (
SEGFAULT)
        1321 - EventBaseLocalTest.EventBaseLocalTest.DestructorStressTest (SEGFA
ULT)
        1341 - NotificationQueueTest.NotificationQueueTest.ConsumeUntilDrained (
SEGFAULT)
        1342 - NotificationQueueTest.NotificationQueueTest.ConsumeUntilDrainedSt
ress (SEGFAULT)
        1343 - NotificationQueueTest.NotificationQueueTest.SendOneEventFD (SEGFA
ULT)
        1344 - NotificationQueueTest.NotificationQueueTest.PutMessagesEventFD (S
EGFAULT)
        1345 - NotificationQueueTest.NotificationQueueTest.MultiConsumerEventFD
(SEGFAULT)
        1349 - NotificationQueueTest.NotificationQueueTest.SendOnePipe (SEGFAULT
)
        1350 - NotificationQueueTest.NotificationQueueTest.PutMessagesPipe (SEGF
AULT)
        1351 - NotificationQueueTest.NotificationQueueTest.MultiConsumerPipe (SE
GFAULT)
        1355 - NotificationQueueTest.NotificationQueueTest.UseAfterFork (SEGFAUL
T)
        1374 - RequestContextTest.RequestContextTest.AccessAllThreadsDestruction
Guard (SEGFAULT)
        1376 - RequestContextTest.RequestContextTryGetTest.TryGetTest (SEGFAULT)
        1379 - ScopedEventBaseThreadTest.ScopedEventBaseThreadTest.example (SEGF
AULT)
        1380 - ScopedEventBaseThreadTest.ScopedEventBaseThreadTest.named_example
 (SEGFAULT)
        1381 - ScopedEventBaseThreadTest.ScopedEventBaseThreadTest.default_manag
er (SEGFAULT)
        1382 - ScopedEventBaseThreadTest.ScopedEventBaseThreadTest.custom_manage
r (SEGFAULT)
        1383 - ScopedEventBaseThreadTest.ScopedEventBaseThreadTest.eb_dtor_in_io
_thread (SEGFAULT)
        1384 - ScopedEventBaseThreadTest.ScopedEventBaseThreadTest.keepalive (SE
GFAULT)
        1387 - writechain_test.WriteChainAsyncTransportWrapperTest.TestSimpleIov
 (SEGFAULT)
        1388 - writechain_test.WriteChainAsyncTransportWrapperTest.TestChainedIo
v (SEGFAULT)
        1389 - writechain_test.WriteChainAsyncTransportWrapperTest.TestSimpleBuf
 (SEGFAULT)
        1696 - baton_test.Baton.pingpongBlocking (SEGFAULT)
        1697 - baton_test.Baton.pingpongNonblocking (SEGFAULT)
        1700 - baton_test.Baton.timedWaitTimeoutSystemClockBlocking (SEGFAULT)
        1701 - baton_test.Baton.timedWaitTimeoutSystemClockNonblocking (SEGFAULT
)
        1702 - baton_test.Baton.timedWaitSystemClockBlocking (SEGFAULT)
        1703 - baton_test.Baton.timedWaitSystemClockNonblocking (SEGFAULT)
        1706 - baton_test.Baton.timedWaitTimeoutSteadyClockBlocking (SEGFAULT)
        1707 - baton_test.Baton.timedWaitTimeoutSteadyClockNonblocking (SEGFAULT
)
        1708 - baton_test.Baton.timedWaitSteadyClockBlocking (SEGFAULT)
        1709 - baton_test.Baton.timedWaitSteadyClockNonblocking (SEGFAULT)
        1721 - lifo_sem_test.LifoSemTest.pingpong (SEGFAULT)
        1722 - lifo_sem_test.LifoSemTest.mutex (SEGFAULT)
        1726 - lifo_sem_test.LifoSemTest.shutdown_multi (SEGFAULT)
        1822 - small_locks_test.SmallLocks.SpinLockCorrectness (SEGFAULT)
        2008 - deterministic_schedule_test.DeterministicSchedule.buggyAdd (SEGFA
ULT)
        2009 - deterministic_schedule_test.DeterministicSchedule.globalInvariant
s (SEGFAULT)
        2178 - file_util_test.WriteFileAtomic.directoryPermissions (Failed)
        2268 - futex_test.Futex.basicDeterministic (SEGFAULT)
        2301 - locks_test.SpinLock.Correctness (SEGFAULT)
        2346 - memory_idler_test.MemoryIdler.futexWaitValueChangedEarly (Timeout
)
        2347 - memory_idler_test.MemoryIdler.futexWaitValueChangedLate (Timeout)
        2348 - memory_idler_test.MemoryIdler.futexWaitAwokenEarly (Timeout)
        2349 - memory_idler_test.MemoryIdler.futexWaitAwokenLate (Timeout)
        2350 - memory_idler_test.MemoryIdler.futexWaitImmediateFlush (Timeout)
        2351 - memory_idler_test.MemoryIdler.futexWaitNeverFlush (Timeout)
        2443 - random_test.Random.MultiThreaded (SEGFAULT)
        2722 - synchronized_test.SynchronizedTimedTest/*.Timed (SEGFAULT)
        2723 - synchronized_test.SynchronizedTimedWithConstTest/*.TimedShared (S
EGFAULT)

And the error msgs are basicly like "signal 11 (SIGSEGV) (0x7fa7220d8e18) received by PID 3988029 (pthread TID 0x 7fa0e901c640) (linux TID 3988056) (code: address not mapped to object)".

I don't know what's wrong and how to fix this...

HolyLow avatar May 27 '24 07:05 HolyLow

I tried to modify the above patch https://github.com/facebook/hhvm/commit/2469b030d57f64a9b39f86b80744237186edea29 to make it turned on for glibc-2.32, and seems many test cases have passed. The failure list shrinks to

The following tests FAILED:
        341 - cache_locality_test.CacheLocality.LinuxActual (Failed)
        1292 - AsyncUDPSocketTest.AsyncUDPSocketTest.TestErrToNonExistentServer (SEGFAULT)
        1293 - AsyncUDPSocketTest.AsyncUDPSocketTest.TestUnsetErrCallback (SEGFAULT)
        1294 - AsyncUDPSocketTest.AsyncUDPSocketTest.CloseInErrorCallback (Timeout)
        1295 - AsyncUDPSocketTest.AsyncUDPSocketTest.TestNonExistentServerNoErrCb (SEGFAULT)
        1300 - AsyncUDPSocketTest.AsyncUDPSocketTest.TestDetachAttach (SEGFAULT)
        1301 - AsyncUDPSocketTest.AsyncUDPSocketTest.TestWriteCmsg (SEGFAULT)
        1302 - AsyncUDPSocketTest.AsyncUDPSocketTest.TestWriteDynamicCmsg (SEGFAULT)
        1304 - AsyncUDPSocketTest.AsyncUDPSocketTest.TestWritemCmsg (SEGFAULT)
        1305 - AsyncUDPSocketTest.AsyncUDPSocketTest.TestWritemDynamicCmsg (SEGFAULT)
        1307 - AsyncUDPSocketTest.AsyncUDPSocketTest.TestWritemNontrivialCmsgs (SEGFAULT)
        1387 - writechain_test.WriteChainAsyncTransportWrapperTest.TestSimpleIov (SEGFAULT)
        1388 - writechain_test.WriteChainAsyncTransportWrapperTest.TestChainedIov (SEGFAULT)
        1389 - writechain_test.WriteChainAsyncTransportWrapperTest.TestSimpleBuf (SEGFAULT)
        2178 - file_util_test.WriteFileAtomic.directoryPermissions (Failed)
        2346 - memory_idler_test.MemoryIdler.futexWaitValueChangedEarly (Timeout)
        2347 - memory_idler_test.MemoryIdler.futexWaitValueChangedLate (Timeout)
        2348 - memory_idler_test.MemoryIdler.futexWaitAwokenEarly (Timeout)
        2349 - memory_idler_test.MemoryIdler.futexWaitAwokenLate (Timeout)
        2350 - memory_idler_test.MemoryIdler.futexWaitImmediateFlush (Timeout)
        2351 - memory_idler_test.MemoryIdler.futexWaitNeverFlush (Timeout)

HolyLow avatar May 27 '24 08:05 HolyLow

UPDATE: The latest main branch of folly has enabled the patch for all GLIBC cases, so the problem should be solved.

HolyLow avatar May 29 '24 02:05 HolyLow