Assertion `false' failed to hold. Replaying 'SIGNAL: SIGCHLD(async)': expecting tracee signal or trap, but instead at 'futex'

[tysmith]

Triggered while running rr replay using rr commit 2bb38b94.

[FATAL src/ReplaySession.cc:684:check_pending_sig()] 
 (task 209395 (rec:208378) at time 158709)
 -> Assertion `false' failed to hold. Replaying `SIGNAL: SIGCHLD(async)': expecting tracee signal or trap, but instead at `futex' (ticks: 1938776)
Tail of trace dump:
{
  real_time:73223.457379 global_time:158689, event:`SYSCALLBUF_RESET' tid:208828, ticks:453157052
}
{
  real_time:73223.457457 global_time:158690, event:`SYSCALL: poll' (state:EXITING_SYSCALL) tid:208394, ticks:929
rax:0x1 rbx:0x7f7bd38fffa0 rcx:0xffffffffffffffff rdx:0xffffffff rsi:0x2 rdi:0x7f7bd4e78d00 rbp:0x7f7bd4e78d40 rsp:0x7f7bd38ffdf0 r8:0x0 r9:0x7 r10:0x1 r11:0x246 r12:0x7f7bd4e79640 r13:0x0 r14:0x7f7bd4e78d00 r15:0x7f7bd4e78cff rip:0x70000002 eflags:0x246 cs:0x33 ss:0x2b ds:0x0 es:0x0 fs:0x0 gs:0x0 orig_rax:0x7 fs_base:0x7f7bd4e79640 gs_base:0x0
  { tid:208394, addr:0x7f7bd4e78d00, length:0x10 }
}
{
  real_time:73223.457519 global_time:158691, event:`SYSCALLBUF_FLUSH' tid:208394, ticks:1003
  { syscall:'recvmsg', ret:0x630, size:0x6c0, desched:1 }
}
{
  real_time:73223.457524 global_time:158692, event:`SYSCALL: rrcall_notify_control_msg' (state:ENTERING_SYSCALL) tid:208394, ticks:1003
rax:0xffffffffffffffda rbx:0x7f7bd38fffa0 rcx:0xffffffffffffffff rdx:0x0 rsi:0x0 rdi:0x7f7bd4e78550 rbp:0x7f7bd4e78ce0 rsp:0x7f7bd38ffe10 r8:0x0 r9:0x0 r10:0x0 r11:0x246 r12:0x630 r13:0x0 r14:0x7f7bd4e78550 r15:0x7f7bd4e78cff rip:0x70000005 eflags:0x246 cs:0x33 ss:0x2b ds:0x0 es:0x0 fs:0x0 gs:0x0 orig_rax:0x3eb fs_base:0x7f7bd4e79640 gs_base:0x0
}
{
  real_time:73223.457528 global_time:158693, event:`SYSCALLBUF_RESET' tid:208394, ticks:1003
}
{
  real_time:73223.459051 global_time:158694, event:`SYSCALL: rrcall_notify_control_msg' (state:EXITING_SYSCALL) tid:208394, ticks:1003
rax:0x0 rbx:0x7f7bd38fffa0 rcx:0xffffffffffffffff rdx:0x0 rsi:0x0 rdi:0x7f7bd4e78550 rbp:0x7f7bd4e78ce0 rsp:0x7f7bd38ffe10 r8:0x0 r9:0x0 r10:0x0 r11:0x246 r12:0x630 r13:0x0 r14:0x7f7bd4e78550 r15:0x7f7bd4e78cff rip:0x70000005 eflags:0x246 cs:0x33 ss:0x2b ds:0x0 es:0x0 fs:0x0 gs:0x0 orig_rax:0x3eb fs_base:0x7f7bd4e79640 gs_base:0x0
}
{
  real_time:73223.459123 global_time:158695, event:`SYSCALL: openat' (state:ENTERING_SYSCALL) tid:208394, ticks:1761
rax:0xffffffffffffffda rbx:0x7f7bd38fffa0 rcx:0xffffffffffffffff rdx:0x80041 rsi:0x7f7bd4e773e8 rdi:0xffffff9c rbp:0x7f7bd4e773e8 rsp:0x7f7bd38ffdf0 r8:0x0 r9:0x3010101070100 r10:0x1b6 r11:0x246 r12:0x80041 r13:0x7f7bd4e78aa0 r14:0x7f7bd4e773e8 r15:0x1b6 rip:0x70000002 eflags:0x246 cs:0x33 ss:0x2b ds:0x0 es:0x0 fs:0x0 gs:0x0 orig_rax:0x101 fs_base:0x7f7bd4e79640 gs_base:0x0
}
{
  real_time:73223.460524 global_time:158696, event:`SYSCALL: openat' (state:EXITING_SYSCALL) tid:208394, ticks:1761
rax:0x3c rbx:0x7f7bd38fffa0 rcx:0xffffffffffffffff rdx:0x80041 rsi:0x7f7bd4e773e8 rdi:0xffffff9c rbp:0x7f7bd4e773e8 rsp:0x7f7bd38ffdf0 r8:0x0 r9:0x3010101070100 r10:0x1b6 r11:0x246 r12:0x80041 r13:0x7f7bd4e78aa0 r14:0x7f7bd4e773e8 r15:0x1b6 rip:0x70000002 eflags:0x246 cs:0x33 ss:0x2b ds:0x0 es:0x0 fs:0x0 gs:0x0 orig_rax:0x101 fs_base:0x7f7bd4e79640 gs_base:0x0
}
{
  real_time:73223.460598 global_time:158697, event:`SYSCALLBUF_FLUSH' tid:208394, ticks:1845
  { syscall:'getrandom', ret:0x10, size:0x20, desched:1 }
}
{
  real_time:73223.460602 global_time:158698, event:`SYSCALL: kill' (state:ENTERING_SYSCALL) tid:208394, ticks:1845
rax:0xffffffffffffffda rbx:0x7f7bd38fffa0 rcx:0xffffffffffffffff rdx:0x0 rsi:0x13 rdi:0x32fbc rbp:0x7f7bd4e76d70 rsp:0x7f7bd38ffdf0 r8:0x0 r9:0x3010101070100 r10:0x1b6 r11:0x246 r12:0x7f7bd4e77328 r13:0x8000000000000000 r14:0x5f5e100 r15:0x0 rip:0x70000002 eflags:0x246 cs:0x33 ss:0x2b ds:0x0 es:0x0 fs:0x0 gs:0x0 orig_rax:0x3e fs_base:0x7f7bd4e79640 gs_base:0x0
}
{
  real_time:73223.460605 global_time:158699, event:`SYSCALLBUF_RESET' tid:208394, ticks:1845
}
{
  real_time:73223.460641 global_time:158700, event:`SYSCALL: kill' (state:EXITING_SYSCALL) tid:208394, ticks:1845
rax:0x0 rbx:0x7f7bd38fffa0 rcx:0xffffffffffffffff rdx:0x0 rsi:0x13 rdi:0x32fbc rbp:0x7f7bd4e76d70 rsp:0x7f7bd38ffdf0 r8:0x0 r9:0x3010101070100 r10:0x1b6 r11:0x246 r12:0x7f7bd4e77328 r13:0x8000000000000000 r14:0x5f5e100 r15:0x0 rip:0x70000002 eflags:0x246 cs:0x33 ss:0x2b ds:0x0 es:0x0 fs:0x0 gs:0x0 orig_rax:0x3e fs_base:0x7f7bd4e79640 gs_base:0x0
}
{
  real_time:73223.460773 global_time:158701, event:`SYSCALLBUF_FLUSH' tid:208394, ticks:6088
  { syscall:'clock_gettime', ret:0x0, size:0x20 }
  { syscall:'openat', ret:0x57, size:0x10, desched:1 }
  { syscall:'readlinkat', ret:0x11, size:0x21 }
  { syscall:'statx', ret:0x0, size:0x110 }
  { syscall:'lseek', ret:0x0, size:0x10 }
  { syscall:'read', ret:0x162, size:0x172, desched:1 }
  { syscall:'read', ret:0x0, size:0x10, desched:1 }
  { syscall:'close', ret:0x0, size:0x10 }
}
{
  real_time:73223.460783 global_time:158702, event:`SYSCALL: clock_nanosleep' (state:ENTERING_SYSCALL) tid:208394, ticks:6088
rax:0xffffffffffffffda rbx:0x1 rcx:0xffffffffffffffff rdx:0x7f7bd4e76d10 rsi:0x0 rdi:0x0 rbp:0x7f7bd4e77270 rsp:0x7f7bd4e76c70 r8:0x0 r9:0x3 r10:0x7f7bd4e76d10 r11:0x246 r12:0x7f7bd4e76d10 r13:0x0 r14:0x7f7bd4e76d10 r15:0x7fffffffffffffff rip:0x7f7bf90e57f8 eflags:0x246 cs:0x33 ss:0x2b ds:0x0 es:0x0 fs:0x0 gs:0x0 orig_rax:0xe6 fs_base:0x7f7bd4e79640 gs_base:0x0
}
{
  real_time:73223.460786 global_time:158703, event:`SYSCALLBUF_RESET' tid:208394, ticks:6088
}
{
  real_time:73223.461028 global_time:158704, event:`SIGNAL: SIGSTOP(async)' tid:208853, ticks:260884711
rax:0x100 rbx:0x3 rcx:0x100 rdx:0x0 rsi:0x88f1375342677800 rdi:0x7f9b4b9a13b8 rbp:0x7f9b4b9a13d0 rsp:0x7f9b4b9a13b0 r8:0x7f9b4b9a1310 r9:0x7f9b483e5a00 r10:0x2 r11:0x537b r12:0x305 r13:0x7f9b48dcc828 r14:0x0 r15:0x7f9b48dcc000 rip:0x7f9b5b53632e eflags:0x206 cs:0x33 ss:0x2b ds:0x0 es:0x0 fs:0x0 gs:0x0 orig_rax:0xffffffffffffffff fs_base:0x7f9b4b9a2640 gs_base:0x0 st0:0x0 st1:0x0 st2:0x0 st3:0x0 st4:0x0 st5:0x0 st6:0x403ade0b6b3a76400012 st7:0x0 ymm0:0xaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa ymm1:0x10000ffffffffffffffff00000000 ymm2:0x7f9b417a92c800007f9b417a9270 ymm3:0x7f9b417a8e3000007f9b417a8dd8 ymm4:0x22f360b05302e2c1e1a16120e0a0806 ymm5:0x400305463442403e3434343c3a343432 ymm6:0x34340a083434322a22940a538e66022f ymm7:0x2a22920a538c66022f360b0536342c1e ymm8:0x22f360b05302e2c1e1a16120e0a0806 ymm9:0x0 ymm10:0x3f800000 ymm11:0x0 ymm12:0x0 ymm13:0x42aa0000 ymm14:0x0 ymm15:0x3f800000
}
{
  real_time:73223.461073 global_time:158705, event:`SIGNAL_DELIVERY: SIGSTOP(async)' tid:208853, ticks:260884711
rax:0x100 rbx:0x3 rcx:0x100 rdx:0x0 rsi:0x88f1375342677800 rdi:0x7f9b4b9a13b8 rbp:0x7f9b4b9a13d0 rsp:0x7f9b4b9a13b0 r8:0x7f9b4b9a1310 r9:0x7f9b483e5a00 r10:0x2 r11:0x537b r12:0x305 r13:0x7f9b48dcc828 r14:0x0 r15:0x7f9b48dcc000 rip:0x7f9b5b53632e eflags:0x206 cs:0x33 ss:0x2b ds:0x0 es:0x0 fs:0x0 gs:0x0 orig_rax:0xffffffffffffffff fs_base:0x7f9b4b9a2640 gs_base:0x0
  { tid:208853, addr:0x7f9b4b9a13b0, length:(nil) }
}
{
  real_time:73223.462370 global_time:158706, event:`SYSCALL: futex' (state:EXITING_SYSCALL) tid:208378, ticks:1938776
rax:0xfffffffffffffe00 rbx:0x7f7bd88fffa0 rcx:0xffffffffffffffff rdx:0xffffffff rsi:0x89 rdi:0x7f7be3f26788 rbp:0x7f7bf6bfead0 rsp:0x7f7bd88ffdf0 r8:0x0 r9:0x7f7bffffffff r10:0x0 r11:0x246 r12:0x7f7bf911e870 r13:0x7f7bf902a250 r14:0x7f7bf6bfea60 r15:0x0 rip:0x70000002 eflags:0x246 cs:0x33 ss:0x2b ds:0x0 es:0x0 fs:0x0 gs:0x0 orig_rax:0xca fs_base:0x7f7bf6bff640 gs_base:0x0
}
{
  real_time:73223.462425 global_time:158707, event:`SIGNAL: SIGCHLD(async)' tid:208378, ticks:1938776
rax:0xfffffffffffffe00 rbx:0x7f7bd88fffa0 rcx:0xffffffffffffffff rdx:0xffffffff rsi:0x89 rdi:0x7f7be3f26788 rbp:0x7f7bf6bfead0 rsp:0x7f7bd88ffdf0 r8:0x0 r9:0x7f7bffffffff r10:0x0 r11:0x246 r12:0x7f7bf911e870 r13:0x7f7bf902a250 r14:0x7f7bf6bfea60 r15:0x0 rip:0x70000002 eflags:0x246 cs:0x33 ss:0x2b ds:0x0 es:0x0 fs:0x0 gs:0x0 orig_rax:0xca fs_base:0x7f7bf6bff640 gs_base:0x0 st0:0x0 st1:0x0 st2:0x0 st3:0x0 st4:0x0 st5:0x0 st6:0x403ade0b6b3a76400012 st7:0x403ade0b6b3a76400000 ymm0:0x0 ymm1:0x7f7bd01764800000000000000001 ymm2:0x7f7bd3afec9000007f7bd01039c0 ymm3:0x7f7bb8496a600000000000000001 ymm4:0x2f80712f098806577aff56b20fb51ba6 ymm5:0x700007f7bb870b400 ymm6:0x60000000000000000 ymm7:0x7ffcefbc9d408000000000000000 ymm8:0x0 ymm9:0x0 ymm10:0x0 ymm11:0x0 ymm12:0x0 ymm13:0x0 ymm14:0x0 ymm15:0x0
}
{
  real_time:73223.462456 global_time:158708, event:`SIGNAL_DELIVERY: SIGCHLD(async)' tid:208378, ticks:1938776
rax:0xca rbx:0x7f7bd88fffa0 rcx:0xffffffffffffffff rdx:0xffffffff rsi:0x89 rdi:0x7f7be3f26788 rbp:0x7f7bf6bfead0 rsp:0x7f7bd88ffdf0 r8:0x0 r9:0x7f7bffffffff r10:0x0 r11:0x246 r12:0x7f7bf911e870 r13:0x7f7bf902a250 r14:0x7f7bf6bfea60 r15:0x0 rip:0x70000000 eflags:0x246 cs:0x33 ss:0x2b ds:0x0 es:0x0 fs:0x0 gs:0x0 orig_rax:0xca fs_base:0x7f7bf6bff640 gs_base:0x0
  { tid:208378, addr:0x7f7bd88ffdf0, length:(nil) }
}
{
  real_time:73223.462615 global_time:158709, event:`SIGNAL: SIGCHLD(async)' tid:208378, ticks:1938776
rax:0xfffffffffffffe00 rbx:0x7f7bd88fffa0 rcx:0xffffffffffffffff rdx:0xffffffff rsi:0x89 rdi:0x7f7be3f26788 rbp:0x7f7bf6bfead0 rsp:0x7f7bd88ffdf0 r8:0x0 r9:0x7f7bffffffff r10:0x0 r11:0x246 r12:0x7f7bf911e870 r13:0x7f7bf902a250 r14:0x7f7bf6bfea60 r15:0x0 rip:0x70000002 eflags:0x246 cs:0x33 ss:0x2b ds:0x0 es:0x0 fs:0x0 gs:0x0 orig_rax:0xca fs_base:0x7f7bf6bff640 gs_base:0x0 st0:0x0 st1:0x0 st2:0x0 st3:0x0 st4:0x0 st5:0x0 st6:0x403ade0b6b3a76400012 st7:0x403ade0b6b3a76400000 ymm0:0x0 ymm1:0x7f7bd01764800000000000000001 ymm2:0x7f7bd3afec9000007f7bd01039c0 ymm3:0x7f7bb8496a600000000000000001 ymm4:0x2f80712f098806577aff56b20fb51ba6 ymm5:0x700007f7bb870b400 ymm6:0x60000000000000000 ymm7:0x7ffcefbc9d408000000000000000 ymm8:0x0 ymm9:0x0 ymm10:0x0 ymm11:0x0 ymm12:0x0 ymm13:0x0 ymm14:0x0 ymm15:0x0
}
{
  real_time:73223.462644 global_time:158710, event:`SIGNAL_DELIVERY: SIGCHLD(async)' tid:208378, ticks:1938776
rax:0xca rbx:0x7f7bd88fffa0 rcx:0xffffffffffffffff rdx:0xffffffff rsi:0x89 rdi:0x7f7be3f26788 rbp:0x7f7bf6bfead0 rsp:0x7f7bd88ffdf0 r8:0x0 r9:0x7f7bffffffff r10:0x0 r11:0x246 r12:0x7f7bf911e870 r13:0x7f7bf902a250 r14:0x7f7bf6bfea60 r15:0x0 rip:0x70000000 eflags:0x246 cs:0x33 ss:0x2b ds:0x0 es:0x0 fs:0x0 gs:0x0 orig_rax:0xca fs_base:0x7f7bf6bff640 gs_base:0x0
  { tid:208378, addr:0x7f7bd88ffdf0, length:(nil) }
}
=== Start rr backtrace:
rr(_ZN2rr13dump_rr_stackEv+0x5a)[0x555bdd16701a]
rr(_ZN2rr15emergency_debugEPNS_4TaskE+0xe8)[0x555bdd03dd48]
rr(+0xd3c0a)[0x555bdd046c0a]
rr(+0xd3e0f)[0x555bdd046e0f]
rr(+0xd3e49)[0x555bdd046e49]
rr(_ZN2rr13ReplaySession17check_pending_sigEPNS_10ReplayTaskE+0xd6)[0x555bdd0d1d06]
rr(_ZN2rr13ReplaySession16continue_or_stepEPNS_10ReplayTaskERKNS0_15StepConstraintsENS_12TicksRequestENS_13ResumeRequestE+0xa3)[0x555bdd0d8db3]
rr(_ZN2rr13ReplaySession20emulate_async_signalEPNS_10ReplayTaskERKNS0_15StepConstraintsElNS_15remote_code_ptrE+0x59e)[0x555bdd0dc46e]
rr(_ZN2rr13ReplaySession18try_one_trace_stepEPNS_10ReplayTaskERKNS0_15StepConstraintsE+0x289)[0x555bdd0e0769]
rr(_ZN2rr13ReplaySession11replay_stepERKNS0_15StepConstraintsE+0x21d)[0x555bdd0e1fad]
rr(_ZN2rr14ReplayTimeline19replay_step_forwardENS_10RunCommandE+0xbf)[0x555bdd0fe9ef]
rr(_ZN2rr9GdbServer14debug_one_stepERNS_10GdbRequestE+0x38b)[0x555bdd02ad7b]
rr(_ZN2rr9GdbServer12serve_replayERKNS0_15ConnectionFlagsE+0xc9b)[0x555bdd02ca0b]
rr(+0x162a97)[0x555bdd0d5a97]
rr(_ZN2rr13ReplayCommand3runERSt6vectorINSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEESaIS7_EE+0x69a)[0x555bdd0d703a]
rr(main+0x1a1)[0x555bdcfc2c61]
/lib/x86_64-linux-gnu/libc.so.6(+0x29d90)[0x7fbcfea29d90]
/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0x80)[0x7fbcfea29e40]
rr(_start+0x25)[0x555bdcfc5635]
=== End rr backtrace

[khuey]

Does replaying the same trace multiple times die in the same way?

Is there anything interesting about the workload (e.g. WebRTC or something)?

What CPU are you running on?

[tysmith]

Does replaying the same trace multiple times die in the same way?

Yes.

Is there anything interesting about the workload (e.g. WebRTC or something)?

It's a live website, I can provide a link if required.

What CPU are you running on?

Intel i7-8650U.

[khuey]

A link and a mozilla-central rev would be good.

[tysmith]

Website is: http://interactive.index.hr/

I was running m-c 96b6835d33ed.

[khuey]

Actually linking me to the exact binaries you were using might be a better idea than me building from source.

[tysmith]

Oh it was a local build sorry.

[tysmith]

relevant mozconfig parts are:

ac_add_options --enable-debug
ac_add_options --disable-install-strip
ac_add_options --enable-optimize="-O1 -g"
export MOZ_DEBUG_SYMBOLS=1
ac_add_options --enable-debug-symbols