rr
rr copied to clipboard
rr-debugger
RR crash on mprotect
rr replay after a rr record:
[FATAL src/AutoRemoteSyscalls.cc:905:check_syscall_result()]
(task 120571 (rec:120521) at time 1749)
-> Assertion `false' failed to hold. Syscall mprotect failed with errno ENOMEM arg1=0x15d300000000 arg2=0x14afffff000 arg3=0x2 arg4=0x32 arg5=0xffffffffffffffff arg6=0x0
Tail of trace dump:
{
real_time:24514.336082 global_time:1729, event:`SYSCALL: ioctl' (state:ENTERING_SYSCALL) tid:120521, ticks:11770025
rax:0xffffffffffffffda rbx:0x681fffa0 rcx:0xffffffffffffffff rdx:0x7ffe9850b560 rsi:0x5401 rdi:0x5 rbp:0x575042d3e800 rsp:0x681ffdd0 r8:0x7ffe9850b5a0 r9:0x1 r10:0x1000 r11:0x246 r12:0x16 r13:0x7ffe9850b7b0 r14:0x7ffe9850b7b0 r15:0x74f02d486a80 rip:0x70000002 eflags:0x246 cs:0x33 ss:0x2b ds:0x0 es:0x0 fs:0x0 gs:0x0 orig_rax:0x10 fs_base:0x74f02d486a80 gs_base:0x0
}
{
real_time:24514.336090 global_time:1730, event:`SYSCALLBUF_RESET' tid:120521, ticks:11770025
}
{
real_time:24514.336118 global_time:1731, event:`SYSCALL: ioctl' (state:EXITING_SYSCALL) tid:120521, ticks:11770025
rax:0xffffffffffffffea rbx:0x681fffa0 rcx:0xffffffffffffffff rdx:0x7ffe9850b560 rsi:0x5401 rdi:0x5 rbp:0x575042d3e800 rsp:0x681ffdd0 r8:0x7ffe9850b5a0 r9:0x1 r10:0x1000 r11:0x246 r12:0x16 r13:0x7ffe9850b7b0 r14:0x7ffe9850b7b0 r15:0x74f02d486a80 rip:0x70000002 eflags:0x246 cs:0x33 ss:0x2b ds:0x0 es:0x0 fs:0x0 gs:0x0 orig_rax:0x10 fs_base:0x74f02d486a80 gs_base:0x0
{ tid:120521, addr:0x7ffe9850b560, length:0x24 }
}
{
real_time:24514.336377 global_time:1732, event:`SYSCALLBUF_FLUSH' tid:120521, ticks:11798804
{ syscall:'read', ret:0x1000, size:0x1010, desched:1 }
{ syscall:'close', ret:0x0, size:0x10 }
{ syscall:'mprotect', ret:0x0, size:0x10 }
{ syscall:'mprotect', ret:0x0, size:0x10 }
{ syscall:'mprotect', ret:0x0, size:0x10 }
{ syscall:'mprotect', ret:0x0, size:0x10 }
{ syscall:'mprotect', ret:0x0, size:0x10 }
{ start:0x74ef9c000000, size:10000, prot:'r--' }
{ start:0x74ef9c010000, size:4000, prot:'rw-' }
{ start:0x74ef9c020000, size:4000, prot:'rw-' }
{ start:0x74ef9c030000, size:4000, prot:'rw-' }
{ start:0x74ef9c040000, size:4000, prot:'rw-' }
}
{
real_time:24514.336383 global_time:1733, event:`SYSCALL: brk' (state:ENTERING_SYSCALL) tid:120521, ticks:11798804
rax:0xffffffffffffffda rbx:0x575042de8000 rcx:0xffffffffffffffff rdx:0x0 rsi:0x74f02d41ac80 rdi:0x575042de8000 rbp:0x74f02d422218 rsp:0x7ffe9850b518 r8:0x0 r9:0x7f r10:0x575042dc6000 r11:0x246 r12:0x575042dc6000 r13:0x575042db6aa0 r14:0x1000 r15:0x114b0 rip:0x74f02d31a82b eflags:0x246 cs:0x33 ss:0x2b ds:0x0 es:0x0 fs:0x0 gs:0x0 orig_rax:0xc fs_base:0x74f02d486a80 gs_base:0x0
}
{
real_time:24514.336389 global_time:1734, event:`SYSCALLBUF_RESET' tid:120521, ticks:11798804
}
{
real_time:24514.336473 global_time:1735, event:`SYSCALL: brk' (state:EXITING_SYSCALL) tid:120521, ticks:11798804
rax:0x575042de8000 rbx:0x575042de8000 rcx:0xffffffffffffffff rdx:0x0 rsi:0x74f02d41ac80 rdi:0x575042de8000 rbp:0x74f02d422218 rsp:0x7ffe9850b518 r8:0x0 r9:0x7f r10:0x575042dc6000 r11:0x246 r12:0x575042dc6000 r13:0x575042db6aa0 r14:0x1000 r15:0x114b0 rip:0x74f02d31a82b eflags:0x246 cs:0x33 ss:0x2b ds:0x0 es:0x0 fs:0x0 gs:0x0 orig_rax:0xc fs_base:0x74f02d486a80 gs_base:0x0
{ map_file:"<ZERO>", addr:0x575042dc6000, length:0x22000, prot_flags:"rw-p", file_offset:0x0, device:0, inode:0, data_file:"", data_offset:0x0, file_size:0x22000 }
}
{
real_time:24514.433730 global_time:1736, event:`SYSCALLBUF_FLUSH' tid:120521, ticks:21403296
{ syscall:'mprotect', ret:0x0, size:0x10 }
{ syscall:'mprotect', ret:0x0, size:0x10 }
{ syscall:'mprotect', ret:0x0, size:0x10 }
{ syscall:'mprotect', ret:0x0, size:0x10 }
{ syscall:'clock_gettime', ret:0x0, size:0x20 }
{ syscall:'clock_gettime', ret:0x0, size:0x20 }
{ start:0x2ad00040000, size:40000, prot:'rw-' }
{ start:0x311d00100000, size:40000, prot:'rw-' }
{ start:0x74f013200000, size:10000, prot:'rw-' }
{ start:0x74f013200000, size:10000, prot:'r--' }
}
{
real_time:24514.433737 global_time:1737, event:`SYSCALL: mmap' (state:ENTERING_SYSCALL) tid:120521, ticks:21403296
rax:0xffffffffffffffda rbx:0x681fffa0 rcx:0xffffffffffffffff rdx:0x0 rsi:0x10000 rdi:0x2ad00010000 rbp:0x7ffe9850b8d0 rsp:0x681ffdf0 r8:0xffffffff r9:0x0 r10:0x32 r11:0x246 r12:0x7ffe9850d008 r13:0x57503e933ee0 r14:0x0 r15:0x74f03a089040 rip:0x70000002 eflags:0x246 cs:0x33 ss:0x2b ds:0x0 es:0x0 fs:0x0 gs:0x0 orig_rax:0x9 fs_base:0x74f02d486a80 gs_base:0x0
}
{
real_time:24514.433742 global_time:1738, event:`SYSCALLBUF_RESET' tid:120521, ticks:21403296
}
{
real_time:24514.433807 global_time:1739, event:`SYSCALL: mmap' (state:EXITING_SYSCALL) tid:120521, ticks:21403296
rax:0x2ad00010000 rbx:0x681fffa0 rcx:0xffffffffffffffff rdx:0x0 rsi:0x10000 rdi:0x2ad00010000 rbp:0x7ffe9850b8d0 rsp:0x681ffdf0 r8:0xffffffff r9:0x0 r10:0x32 r11:0x246 r12:0x7ffe9850d008 r13:0x57503e933ee0 r14:0x0 r15:0x74f03a089040 rip:0x70000002 eflags:0x246 cs:0x33 ss:0x2b ds:0x0 es:0x0 fs:0x0 gs:0x0 orig_rax:0x9 fs_base:0x74f02d486a80 gs_base:0x0
{ map_file:"<ZERO>", addr:0x2ad00010000, length:0x10000, prot_flags:"---p", file_offset:0x0, device:0, inode:0, data_file:"", data_offset:0x0, file_size:0x10000 }
}
{
real_time:24514.434290 global_time:1740, event:`SYSCALLBUF_FLUSH' tid:120521, ticks:21482591
{ syscall:'clock_gettime', ret:0x0, size:0x20 }
{ syscall:'clock_gettime', ret:0x0, size:0x20 }
{ syscall:'openat', ret:0x5, size:0x10, desched:1 }
{ syscall:'readlinkat', ret:0xc, size:0x1c }
{ syscall:'fstatat', ret:0x0, size:0xa0 }
}
{
real_time:24514.434298 global_time:1741, event:`SYSCALL: ioctl' (state:ENTERING_SYSCALL) tid:120521, ticks:21482591
rax:0xffffffffffffffda rbx:0x681fffa0 rcx:0xffffffffffffffff rdx:0x7ffe9850b1e0 rsi:0x5401 rdi:0x5 rbp:0x575042d3e800 rsp:0x681ffdd0 r8:0x7ffe9850b220 r9:0x1 r10:0x1000 r11:0x246 r12:0x16 r13:0x7ffe9850b430 r14:0x7ffe9850b430 r15:0x74f02d486a80 rip:0x70000002 eflags:0x246 cs:0x33 ss:0x2b ds:0x0 es:0x0 fs:0x0 gs:0x0 orig_rax:0x10 fs_base:0x74f02d486a80 gs_base:0x0
}
{
real_time:24514.434303 global_time:1742, event:`SYSCALLBUF_RESET' tid:120521, ticks:21482591
}
{
real_time:24514.434335 global_time:1743, event:`SYSCALL: ioctl' (state:EXITING_SYSCALL) tid:120521, ticks:21482591
rax:0xffffffffffffffea rbx:0x681fffa0 rcx:0xffffffffffffffff rdx:0x7ffe9850b1e0 rsi:0x5401 rdi:0x5 rbp:0x575042d3e800 rsp:0x681ffdd0 r8:0x7ffe9850b220 r9:0x1 r10:0x1000 r11:0x246 r12:0x16 r13:0x7ffe9850b430 r14:0x7ffe9850b430 r15:0x74f02d486a80 rip:0x70000002 eflags:0x246 cs:0x33 ss:0x2b ds:0x0 es:0x0 fs:0x0 gs:0x0 orig_rax:0x10 fs_base:0x74f02d486a80 gs_base:0x0
{ tid:120521, addr:0x7ffe9850b1e0, length:0x24 }
}
{
real_time:24514.560057 global_time:1744, event:`SYSCALLBUF_FLUSH' tid:120521, ticks:35548633
{ syscall:'read', ret:0x1000, size:0x1010, desched:1 }
{ syscall:'close', ret:0x0, size:0x10 }
{ syscall:'mprotect', ret:0x0, size:0x10 }
{ syscall:'clock_gettime', ret:0x0, size:0x20 }
{ syscall:'clock_gettime', ret:0x0, size:0x20 }
{ syscall:'clock_gettime', ret:0x0, size:0x20 }
{ start:0x2ad00180000, size:40000, prot:'rw-' }
}
{
real_time:24514.560068 global_time:1745, event:`SYSCALL: mmap' (state:ENTERING_SYSCALL) tid:120521, ticks:35548633
rax:0xffffffffffffffda rbx:0x681fffa0 rcx:0xffffffffffffffff rdx:0x3 rsi:0x44000 rdi:0x0 rbp:0x43270 rsp:0x681ffdf0 r8:0xffffffff r9:0x0 r10:0x22 r11:0x246 r12:0x0 r13:0x43270 r14:0x1000 r15:0x4325 rip:0x70000002 eflags:0x246 cs:0x33 ss:0x2b ds:0x0 es:0x0 fs:0x0 gs:0x0 orig_rax:0x9 fs_base:0x74f02d486a80 gs_base:0x0
}
{
real_time:24514.560074 global_time:1746, event:`SYSCALLBUF_RESET' tid:120521, ticks:35548633
}
{
real_time:24514.560120 global_time:1747, event:`SYSCALL: mmap' (state:EXITING_SYSCALL) tid:120521, ticks:35548633
rax:0x74f02cdbc000 rbx:0x681fffa0 rcx:0xffffffffffffffff rdx:0x3 rsi:0x44000 rdi:0x0 rbp:0x43270 rsp:0x681ffdf0 r8:0xffffffff r9:0x0 r10:0x22 r11:0x246 r12:0x0 r13:0x43270 r14:0x1000 r15:0x4325 rip:0x70000002 eflags:0x246 cs:0x33 ss:0x2b ds:0x0 es:0x0 fs:0x0 gs:0x0 orig_rax:0x9 fs_base:0x74f02d486a80 gs_base:0x0
{ map_file:"<ZERO>", addr:0x74f02cdbc000, length:0x44000, prot_flags:"rw-p", file_offset:0x0, device:0, inode:0, data_file:"", data_offset:0x0, file_size:0x44000 }
}
{
real_time:24514.560307 global_time:1748, event:`SYSCALL: mmap' (state:ENTERING_SYSCALL) tid:120521, ticks:35583498
rax:0xffffffffffffffda rbx:0x681fffa0 rcx:0xffffffffffffffff rdx:0x0 rsi:0x14afffff000 rdi:0x15d300000000 rbp:0x7ffe9850c090 rsp:0x681ffdf0 r8:0x4 r9:0x0 r10:0x2 r11:0x246 r12:0x7ffe9850d008 r13:0x57503e933ee0 r14:0x0 r15:0x74f03a089040 rip:0x70000002 eflags:0x246 cs:0x33 ss:0x2b ds:0x0 es:0x0 fs:0x0 gs:0x0 orig_rax:0x9 fs_base:0x74f02d486a80 gs_base:0x0
}
{
real_time:24514.561612 global_time:1749, event:`SYSCALL: mmap' (state:EXITING_SYSCALL) tid:120521, ticks:35583498
rax:0x15d300000000 rbx:0x681fffa0 rcx:0xffffffffffffffff rdx:0x0 rsi:0x14afffff000 rdi:0x15d300000000 rbp:0x7ffe9850c090 rsp:0x681ffdf0 r8:0x4 r9:0x0 r10:0x2 r11:0x246 r12:0x7ffe9850d008 r13:0x57503e933ee0 r14:0x0 r15:0x74f03a089040 rip:0x70000002 eflags:0x246 cs:0x33 ss:0x2b ds:0x0 es:0x0 fs:0x0 gs:0x0 orig_rax:0x9 fs_base:0x74f02d486a80 gs_base:0x0
{ map_file:"/memfd:V8MemFDForTesting (deleted)", addr:0x15d300000000, length:0x14afffff000, prot_flags:"---p", file_offset:0x0, device:1, inode:98867, data_file:"", data_offset:0x0, file_size:0x14a00000000 }
{ tid:120521, addr:0x15d300000000, length:0x14a00000000, holes:[(nil)-0x9000000000x900200000-0x14a00000000] }
}
{
real_time:24514.561656 global_time:1750, event:`SYSCALL: munmap' (state:ENTERING_SYSCALL) tid:120521, ticks:35583533
rax:0xffffffffffffffda rbx:0x681fffa0 rcx:0xffffffffffffffff rdx:0x74f039fa8667 rsi:0xfffff000 rdi:0x171d00000000 rbp:0x7ffe9850c090 rsp:0x681ffdf0 r8:0x4 r9:0x0 r10:0x2 r11:0x246 r12:0x7ffe9850d008 r13:0x57503e933ee0 r14:0x0 r15:0x74f03a089040 rip:0x70000002 eflags:0x246 cs:0x33 ss:0x2b ds:0x0 es:0x0 fs:0x0 gs:0x0 orig_rax:0xb fs_base:0x74f02d486a80 gs_base:0x0
}
=== Start rr backtrace:
rr(_ZN2rr13dump_rr_stackERNS_8ScopedFdE+0x5b)[0x5e1291746a0b]
rr(_ZN2rr15emergency_debugEPNS_4TaskE+0x152)[0x5e1291619f12]
rr(+0xdfa82)[0x5e129161aa82]
rr(+0xe008f)[0x5e129161b08f]
rr(_ZN2rr18AutoRemoteSyscalls20check_syscall_resultElib+0x4ae)[0x5e12915ac64e]
rr(+0x1db393)[0x5e1291716393]
rr(_ZN2rr4Task35write_bytes_helper_no_notificationsENS_10remote_ptrIvEElPKvPbj+0x148)[0x5e1291719698]
rr(_ZN2rr4Task18write_bytes_helperENS_10remote_ptrIvEElPKvPbj+0x20)[0x5e12917199b0]
rr(+0x18cc3c)[0x5e12916c7c3c]
rr(_ZN2rr10ReplayTask28apply_data_record_from_traceEv+0xae)[0x5e12916c7f0e]
rr(+0x1887da)[0x5e12916c37da]
rr(+0x18ef7f)[0x5e12916c9f7f]
rr(_ZN2rr13ReplaySession28setup_replay_one_trace_frameEPNS_10ReplayTaskE+0x994)[0x5e12916bc8b4]
rr(_ZN2rr13ReplaySession11replay_stepERKNS0_15StepConstraintsE+0xf9)[0x5e12916bcb19]
rr(_ZN2rr14ReplayTimeline19replay_step_forwardENS_10RunCommandE+0xbf)[0x5e12916d9aef]
rr(_ZN2rr9GdbServer14debug_one_stepERNS_10GdbRequestE+0xf48)[0x5e12915fe5b8]
rr(_ZN2rr9GdbServer12serve_replayESt10shared_ptrINS_13ReplaySessionEERKNS0_6TargetEPVbNS_12DebuggerTypeERKNS0_15ConnectionFlagsE+0x977)[0x5e12915ff3b7]
rr(_ZN2rr13ReplayCommand3runERSt6vectorINSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEESaIS7_EE+0xd18)[0x5e12916af708]
rr(main+0x1a1)[0x5e1291590441]
/lib/x86_64-linux-gnu/libc.so.6(+0x29d90)[0x799f10e29d90]
/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0x80)[0x799f10e29e40]
rr(_start+0x25)[0x5e1291592f25]
=== End rr backtrace
Launch debugger with
'gdb' '-l' '10000' '-ex' 'set sysroot /' '-ex' 'target extended-remote 127.0.0.1:55035' '/home/dmitry/.local/share/rr/cloudflare_main-82/mmap_hardlink_4_cloudflare_main'
Can you dump /proc/<pid>/maps for the replayed tracee process (120571 in the log above)?
Also if you can produce a reduced testcase, or just steps to reproduce, that would be super helpful.
Sure,
90643: [rr:cloudflare_m]
Address Perm Offset Device Inode Size Rss Pss Pss_Dirty Referenced Anonymous KSM LazyFree ShmemPmdMapped FilePmdMapped Shared_Hugetlb Private_Hugetlb Swap SwapPss Locked THPeligible Mapping
68000000 rwxp 00000000 00:00 0 2048 8 8 8 8 8 0 0 0 0 0 0 0 0 0 0
6fffd000 r-xp 00000000 103:05 919164 12 12 12 4 12 4 0 0 0 0 0 0 0 0 0 0 librrpage.so
70000000 r-xp 00004000 103:05 919164 4 4 4 4 4 4 0 0 0 0 0 0 0 0 0 0 librrpage.so
70001000 rw-s 00000000 103:05 5245985 4 4 2 2 4 0 0 0 0 0 0 0 0 0 0 0 rr-shared-preload_thread_locals-90643-0 (deleted)
70002000 rw-s 00000000 103:05 5250827 1024 64 29 5 64 0 0 0 0 0 0 0 0 0 0 0 rr-shared-syscallbuf.89587-90643-3 (deleted)
70102000 rw-s 00000000 103:05 5250834 1024 64 32 2 64 0 0 0 0 0 0 0 0 0 0 0 rr-shared-syscallbuf.89588-90916-5 (deleted)
70203000 r-xp 00000000 00:00 0 4 4 4 4 4 4 0 0 0 0 0 0 0 0 0 0
2a000000000 ---p 00000000 00:00 0 33554432 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
35300000000 ---s 00000000 00:01 167302 37748736 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 memfd:V8MemFDForTesting (deleted) (deleted)
35c00000000 r--s 900000000 00:01 167302 64 32 32 32 32 0 0 0 0 0 0 0 0 0 0 0 memfd:V8MemFDForTesting (deleted) (deleted)
35c00010000 ---p 00000000 00:00 0 64 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
35c00020000 r--s 900020000 00:01 167302 128 128 128 128 128 0 0 0 0 0 0 0 0 0 0 0 memfd:V8MemFDForTesting (deleted) (deleted)
35c00040000 rw-s 900040000 00:01 167302 256 124 124 124 124 0 0 0 0 0 0 0 0 0 0 0 memfd:V8MemFDForTesting (deleted) (deleted)
35c00080000 r--s 900080000 00:01 167302 1024 1024 1024 1024 1024 0 0 0 0 0 0 0 0 0 0 0 memfd:V8MemFDForTesting (deleted) (deleted)
35c00180000 rw-s 900180000 00:01 167302 256 12 12 12 12 0 0 0 0 0 0 0 0 0 0 0 memfd:V8MemFDForTesting (deleted) (deleted)
35c001c0000 r--s 9001c0000 00:01 167302 208 208 208 208 208 0 0 0 0 0 0 0 0 0 0 0 memfd:V8MemFDForTesting (deleted) (deleted)
35c001f4000 ---s 9001f4000 00:01 167302 1346369584 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 memfd:V8MemFDForTesting (deleted) (deleted)
282a00000000 ---p 00000000 00:00 0 1024 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
282a00100000 rw-p 00000000 00:00 0 256 4 4 4 4 4 0 0 0 0 0 0 0 0 0 0
282a00140000 ---p 00000000 00:00 0 1047296 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
60a566278000 r--p 00000000 103:06 3420159 132 96 86 0 96 0 0 0 0 0 0 0 0 0 0 0 mmap_hardlink_4_cloudflare_main
60a566299000 r-xp 00020000 103:06 3420159 172 172 170 0 172 0 0 0 0 0 0 0 0 0 0 0 mmap_hardlink_4_cloudflare_main
60a5662c4000 r--p 0004a000 103:06 3420159 4 4 2 2 4 4 0 0 0 0 0 0 0 0 0 0 mmap_hardlink_4_cloudflare_main
60a5662c5000 rw-p 0004a000 103:06 3420159 4 4 4 4 4 4 0 0 0 0 0 0 0 0 0 0 mmap_hardlink_4_cloudflare_main
60a593367000 rw-p 00000000 00:00 0 688 684 682 682 684 684 0 0 0 0 0 0 0 0 0 0
7399d8001000 ---p 00000000 00:00 0 1388314620 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
74e4d8000000 r--p 00000000 00:00 0 64 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
74e4d8010000 rw-p 00000000 00:00 0 16 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
74e4d8014000 ---p 00000000 00:00 0 48 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
74e4d8020000 rw-p 00000000 00:00 0 16 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
74e4d8024000 ---p 00000000 00:00 0 48 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
74e4d8030000 rw-p 00000000 00:00 0 16 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
74e4d8034000 ---p 00000000 00:00 0 48 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
74e4d8040000 rw-p 00000000 00:00 0 16 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
74e4d8044000 ---p 00000000 00:00 0 65264 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
74e4dc000000 r--p 00000000 00:00 0 64 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
74e4dc010000 rw-p 00000000 00:00 0 16 16 16 16 16 16 0 0 0 0 0 0 0 0 0 0
74e4dc014000 ---p 00000000 00:00 0 48 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
74e4dc020000 rw-p 00000000 00:00 0 16 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
74e4dc024000 ---p 00000000 00:00 0 48 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
74e4dc030000 rw-p 00000000 00:00 0 16 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
74e4dc034000 ---p 00000000 00:00 0 48 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
74e4dc040000 rw-p 00000000 00:00 0 16 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
74e4dc044000 ---p 00000000 00:00 0 524016 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
74e4fc000000 r--p 00000000 00:00 0 64 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
74e4fc010000 rw-p 00000000 00:00 0 16 16 16 16 16 16 0 0 0 0 0 0 0 0 0 0
74e4fc014000 ---p 00000000 00:00 0 48 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
74e4fc020000 rw-p 00000000 00:00 0 16 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
74e4fc024000 ---p 00000000 00:00 0 48 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
74e4fc030000 rw-p 00000000 00:00 0 16 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
74e4fc034000 ---p 00000000 00:00 0 48 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
74e4fc040000 rw-p 00000000 00:00 0 16 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
74e4fc044000 ---p 00000000 00:00 0 65264 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
74e500000000 r--p 00000000 00:00 0 64 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
74e500010000 rw-p 00000000 00:00 0 16 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
74e500014000 ---p 00000000 00:00 0 48 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
74e500020000 rw-p 00000000 00:00 0 16 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
74e500024000 ---p 00000000 00:00 0 48 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
74e500030000 rw-p 00000000 00:00 0 16 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
74e500034000 ---p 00000000 00:00 0 48 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
74e500040000 rw-p 00000000 00:00 0 16 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
74e500044000 ---p 00000000 00:00 0 524016 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
74e520000000 r--p 00000000 00:00 0 64 16 16 16 16 16 0 0 0 0 0 0 0 0 0 0
74e520010000 rw-p 00000000 00:00 0 16 16 16 16 16 16 0 0 0 0 0 0 0 0 0 0
74e520014000 ---p 00000000 00:00 0 48 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
74e520020000 rw-p 00000000 00:00 0 16 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
74e520024000 ---p 00000000 00:00 0 48 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
74e520030000 rw-p 00000000 00:00 0 16 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
74e520034000 ---p 00000000 00:00 0 48 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
74e520040000 rw-p 00000000 00:00 0 16 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
74e520044000 ---p 00000000 00:00 0 524016 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
74e540000000 rwxp 00000000 00:00 0 524288 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
74e560000000 ---p 00000000 00:00 0 131072 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
74e568000000 rw-p 00000000 00:00 0 132 4 4 4 4 4 0 0 0 0 0 0 0 0 0 0
74e568021000 ---p 00000000 00:00 0 65404 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
74e56e800000 r--p 00000000 00:00 0 64 16 16 16 16 16 0 0 0 0 0 0 0 0 0 0
74e56e810000 rw-p 00000000 00:00 0 16 16 16 16 16 16 0 0 0 0 0 0 0 0 0 0
74e56e814000 ---p 00000000 00:00 0 48 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
74e56e820000 rw-p 00000000 00:00 0 16 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
74e56e824000 ---p 00000000 00:00 0 48 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
74e56e830000 rw-p 00000000 00:00 0 16 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
74e56e834000 ---p 00000000 00:00 0 48 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
74e56e840000 rw-p 00000000 00:00 0 16 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
74e56e844000 ---p 00000000 00:00 0 261872 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
74e57e800000 r--p 00000000 00:00 0 64 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
74e57e810000 rw-p 00000000 00:00 0 16 16 16 16 16 16 0 0 0 0 0 0 0 0 0 0
74e57e814000 ---p 00000000 00:00 0 48 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
74e57e820000 rw-p 00000000 00:00 0 16 16 16 16 16 16 0 0 0 0 0 0 0 0 0 0
74e57e824000 ---p 00000000 00:00 0 48 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
74e57e830000 rw-p 00000000 00:00 0 16 16 16 16 16 16 0 0 0 0 0 0 0 0 0 0
74e57e834000 ---p 00000000 00:00 0 48 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
74e57e840000 rw-p 00000000 00:00 0 16 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
74e57e844000 ---p 00000000 00:00 0 130800 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
74e586800000 rwxp 00000000 00:00 0 2048 4 4 4 4 4 0 0 0 0 0 0 0 0 0 0
74e586a83000 ---p 00000000 00:00 0 4 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
74e586a84000 rw-p 00000000 00:00 0 8192 8 8 8 8 8 0 0 0 0 0 0 0 0 0 0
74e587284000 rw-p 00000000 00:00 0 11760 11760 5880 5880 11760 11760 0 0 0 0 0 0 0 0 0 0
74e587e00000 r--p 00000000 103:06 3413588 976 636 636 0 636 0 0 0 0 0 0 0 0 0 0 0 mmap_hardlink_63_libicuuc.so
74e587ef4000 r-xp 000f3000 103:06 3413588 1304 776 776 0 776 0 0 0 0 0 0 0 0 0 0 0 mmap_hardlink_63_libicuuc.so
74e58803a000 r--p 00238000 103:06 3413588 64 64 32 32 64 64 0 0 0 0 0 0 0 0 0 0 mmap_hardlink_63_libicuuc.so
74e58804a000 rw-p 00247000 103:06 3413588 16 16 16 16 16 16 0 0 0 0 0 0 0 0 0 0 mmap_hardlink_63_libicuuc.so
74e58804e000 rw-p 00000000 00:00 0 8 8 8 8 8 8 0 0 0 0 0 0 0 0 0 0
74e588070000 rw-p 00000000 00:00 0 16 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
74e588074000 ---p 00000000 00:00 0 496 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
74e5880f0000 rw-p 00000000 00:00 0 1088 1040 548 548 1040 1040 0 0 0 0 0 0 0 0 0 0 [anon:absl]
74e588200000 r--p 00000000 103:06 3414205 1392 828 828 0 828 0 0 0 0 0 0 0 0 0 0 0 mmap_hardlink_58_libthird_party_icu_icui18n.so
74e58835c000 r-xp 0015b000 103:06 3414205 2128 124 124 0 124 0 0 0 0 0 0 0 0 0 0 0 mmap_hardlink_58_libthird_party_icu_icui18n.so
74e588570000 r--p 0036e000 103:06 3414205 68 68 34 34 68 68 0 0 0 0 0 0 0 0 0 0 mmap_hardlink_58_libthird_party_icu_icui18n.so
74e588581000 rw-p 0037e000 103:06 3414205 28 28 14 14 28 28 0 0 0 0 0 0 0 0 0 0 mmap_hardlink_58_libthird_party_icu_icui18n.so
74e588588000 rw-p 00000000 00:00 0 4 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
74e588600000 r--p 00000000 103:06 3410992 1188 824 824 0 824 0 0 0 0 0 0 0 0 0 0 0 mmap_hardlink_53_libthird_party_abseil-cpp_absl.so
74e588729000 r-xp 00128000 103:06 3410992 1232 1004 1004 4 1004 4 0 0 0 0 0 0 0 0 0 0 mmap_hardlink_53_libthird_party_abseil-cpp_absl.so
74e58885d000 r--p 0025b000 103:06 3410992 12 12 6 6 12 12 0 0 0 0 0 0 0 0 0 0 mmap_hardlink_53_libthird_party_abseil-cpp_absl.so
74e588860000 rw-p 0025d000 103:06 3410992 32 32 22 22 32 32 0 0 0 0 0 0 0 0 0 0 mmap_hardlink_53_libthird_party_abseil-cpp_absl.so
74e588868000 rw-p 00000000 00:00 0 12 4 4 4 4 4 0 0 0 0 0 0 0 0 0 0
74e588891000 rw-p 00000000 00:00 0 544 544 544 544 544 544 0 0 0 0 0 0 0 0 0 0
74e588919000 r--p 00000000 103:05 790465 56 56 0 0 56 0 0 0 0 0 0 0 0 0 0 0 libm.so.6
74e588927000 r-xp 0000e000 103:05 790465 496 284 8 0 284 0 0 0 0 0 0 0 0 0 0 0 libm.so.6
74e5889a3000 r--p 0008a000 103:05 790465 364 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 libm.so.6
74e5889fe000 r--p 000e4000 103:05 790465 4 4 2 2 4 4 0 0 0 0 0 0 0 0 0 0 libm.so.6
74e5889ff000 rw-p 000e5000 103:05 790465 4 4 2 2 4 4 0 0 0 0 0 0 0 0 0 0 libm.so.6
74e588a00000 r--p 00000000 103:05 790438 160 160 1 0 160 0 0 0 0 0 0 0 0 0 0 0 libc.so.6
74e588a28000 r-xp 00028000 103:05 790438 1620 1096 56 48 1096 64 0 0 0 0 0 0 0 0 0 0 libc.so.6
74e588bbd000 r--p 001bd000 103:05 790438 352 188 1 0 188 0 0 0 0 0 0 0 0 0 0 0 libc.so.6
74e588c15000 ---p 00215000 103:05 790438 4 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 libc.so.6
74e588c16000 r--p 00215000 103:05 790438 16 16 8 8 16 16 0 0 0 0 0 0 0 0 0 0 libc.so.6
74e588c1a000 rw-p 00219000 103:05 790438 8 8 8 8 8 8 0 0 0 0 0 0 0 0 0 0 libc.so.6
74e588c1c000 rw-p 00000000 00:00 0 52 20 16 16 20 20 0 0 0 0 0 0 0 0 0 0
74e588c2a000 r-xp 00000000 00:00 0 4 4 4 4 4 4 0 0 0 0 0 0 0 0 0 0
74e588c2d000 r--p 00000000 103:06 3409859 820 456 456 0 456 0 0 0 0 0 0 0 0 0 0 0 mmap_hardlink_39_libc++.so
74e588cfa000 r-xp 000cc000 103:06 3409859 964 792 792 0 792 0 0 0 0 0 0 0 0 0 0 0 mmap_hardlink_39_libc++.so
74e588deb000 r--p 001bc000 103:06 3409859 44 44 22 22 44 44 0 0 0 0 0 0 0 0 0 0 mmap_hardlink_39_libc++.so
74e588df6000 rw-p 001c6000 103:06 3409859 12 12 10 10 12 12 0 0 0 0 0 0 0 0 0 0 mmap_hardlink_39_libc++.so
74e588df9000 rw-p 00000000 00:00 0 28 12 8 8 12 12 0 0 0 0 0 0 0 0 0 0
74e588e00000 r--p 00000000 103:06 3431207 106488 49280 49280 0 49280 0 0 0 0 0 0 0 0 0 0 0 mmap_hardlink_25_libv8.so
74e58f5fe000 ---p 067fe000 103:06 3431207 4 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 mmap_hardlink_25_libv8.so
74e58f5ff000 r-xp 067fe000 103:06 3431207 94344 39432 36902 4 39432 4 0 0 0 0 0 0 0 0 0 0 mmap_hardlink_25_libv8.so
74e595221000 r--p 0c41f000 103:06 3431207 948 948 474 474 948 948 0 0 0 0 0 0 0 0 0 0 mmap_hardlink_25_libv8.so
74e59530e000 ---p 0c50e000 103:06 3431207 4 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 mmap_hardlink_25_libv8.so
74e59530f000 rw-p 0c50c000 103:06 3431207 44 44 44 44 44 44 0 0 0 0 0 0 0 0 0 0 mmap_hardlink_25_libv8.so
74e59531a000 r--p 0c517000 103:06 3431207 4 4 2 2 4 4 0 0 0 0 0 0 0 0 0 0 mmap_hardlink_25_libv8.so
74e59531b000 rw-p 0c518000 103:06 3431207 2300 2300 1544 1544 2300 2300 0 0 0 0 0 0 0 0 0 0 mmap_hardlink_25_libv8.so
74e59555a000 rw-p 00000000 00:00 0 148 44 38 38 44 44 0 0 0 0 0 0 0 0 0 0
74e5955ae000 rw-p 00000000 00:00 0 20 12 10 10 12 12 0 0 0 0 0 0 0 0 0 0
74e5955b3000 r--p 00000000 103:05 789155 12 12 0 0 12 0 0 0 0 0 0 0 0 0 0 0 libgcc_s.so.1
74e5955b6000 r-xp 00003000 103:05 789155 92 64 1 0 64 0 0 0 0 0 0 0 0 0 0 0 libgcc_s.so.1
74e5955cd000 r--p 0001a000 103:05 789155 16 16 0 0 16 0 0 0 0 0 0 0 0 0 0 0 libgcc_s.so.1
74e5955d1000 r--p 0001d000 103:05 789155 4 4 2 2 4 4 0 0 0 0 0 0 0 0 0 0 libgcc_s.so.1
74e5955d2000 rw-p 0001e000 103:05 789155 4 4 2 2 4 4 0 0 0 0 0 0 0 0 0 0 libgcc_s.so.1
74e5955d3000 r--p 00000000 103:06 3410001 56 56 54 0 56 0 0 0 0 0 0 0 0 0 0 0 mmap_hardlink_69_libchrome_zlib.so
74e5955e1000 r-xp 0000d000 103:06 3410001 112 108 106 0 108 0 0 0 0 0 0 0 0 0 0 0 mmap_hardlink_69_libchrome_zlib.so
74e5955fd000 r--p 00028000 103:06 3410001 8 8 4 4 8 8 0 0 0 0 0 0 0 0 0 0 mmap_hardlink_69_libchrome_zlib.so
74e5955ff000 rw-p 00029000 103:06 3410001 4 4 4 4 4 4 0 0 0 0 0 0 0 0 0 0 mmap_hardlink_69_libchrome_zlib.so
74e595600000 rw-p 00000000 00:00 0 8 8 4 4 8 8 0 0 0 0 0 0 0 0 0 0
74e595602000 r--p 00000000 103:05 791612 4 4 0 0 4 0 0 0 0 0 0 0 0 0 0 0 libpthread.so.0
74e595603000 r-xp 00001000 103:05 791612 4 4 0 0 4 0 0 0 0 0 0 0 0 0 0 0 libpthread.so.0
74e595604000 r--p 00002000 103:05 791612 4 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 libpthread.so.0
74e595605000 r--p 00002000 103:05 791612 4 4 2 2 4 4 0 0 0 0 0 0 0 0 0 0 libpthread.so.0
74e595606000 rw-p 00003000 103:05 791612 4 4 2 2 4 4 0 0 0 0 0 0 0 0 0 0 libpthread.so.0
74e595607000 rw-p 00000000 00:00 0 8 8 4 4 8 8 0 0 0 0 0 0 0 0 0 0
74e595609000 r--p 00000000 103:06 3431534 212 156 156 0 156 0 0 0 0 0 0 0 0 0 0 0 mmap_hardlink_34_libv8_libplatform.so
74e59563e000 r-xp 00034000 103:06 3431534 184 184 184 0 184 0 0 0 0 0 0 0 0 0 0 0 mmap_hardlink_34_libv8_libplatform.so
74e59566c000 r--p 00061000 103:06 3431534 4 4 2 2 4 4 0 0 0 0 0 0 0 0 0 0 mmap_hardlink_34_libv8_libplatform.so
74e59566d000 rw-p 00061000 103:06 3431534 12 12 12 12 12 12 0 0 0 0 0 0 0 0 0 0 mmap_hardlink_34_libv8_libplatform.so
74e595670000 r--p 00000000 103:06 3434848 248 192 192 0 192 0 0 0 0 0 0 0 0 0 0 0 mmap_hardlink_30_libv8_libbase.so
74e5956ae000 r-xp 0003d000 103:06 3434848 272 260 260 0 260 0 0 0 0 0 0 0 0 0 0 0 mmap_hardlink_30_libv8_libbase.so
74e5956f2000 r--p 00080000 103:06 3434848 8 8 4 4 8 8 0 0 0 0 0 0 0 0 0 0 mmap_hardlink_30_libv8_libbase.so
74e5956f4000 rw-p 00081000 103:06 3434848 12 12 12 12 12 12 0 0 0 0 0 0 0 0 0 0 mmap_hardlink_30_libv8_libbase.so
74e5956f7000 r--p 00000000 103:05 790462 4 4 0 0 4 0 0 0 0 0 0 0 0 0 0 0 libdl.so.2
74e5956f8000 r-xp 00001000 103:05 790462 4 4 0 0 4 0 0 0 0 0 0 0 0 0 0 0 libdl.so.2
74e5956f9000 r--p 00002000 103:05 790462 4 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 libdl.so.2
74e5956fa000 r--p 00002000 103:05 790462 4 4 2 2 4 4 0 0 0 0 0 0 0 0 0 0 libdl.so.2
74e5956fb000 rw-p 00003000 103:05 790462 4 4 2 2 4 4 0 0 0 0 0 0 0 0 0 0 libdl.so.2
74e595701000 rw-p 00000000 00:00 0 128 8 8 8 8 8 0 0 0 0 0 0 0 0 0 0 [anon:absl]
74e595721000 r--p 00000000 103:05 919163 4 4 4 0 4 0 0 0 0 0 0 0 0 0 0 0 librrpreload.so
74e595722000 r-xp 00001000 103:05 919163 32 32 32 0 32 0 0 0 0 0 0 0 0 0 0 0 librrpreload.so
74e59572a000 r--p 00009000 103:05 919163 12 12 12 0 12 0 0 0 0 0 0 0 0 0 0 0 librrpreload.so
74e59572d000 r--p 0000b000 103:05 919163 4 4 2 2 4 4 0 0 0 0 0 0 0 0 0 0 librrpreload.so
74e59572e000 rw-p 0000c000 103:05 919163 4 4 4 4 4 4 0 0 0 0 0 0 0 0 0 0 librrpreload.so
74e59572f000 rw-p 00000000 00:00 0 48 28 16 16 28 28 0 0 0 0 0 0 0 0 0 0
74e59573b000 r--p 00000000 103:05 790321 8 8 0 0 8 0 0 0 0 0 0 0 0 0 0 0 ld-linux-x86-64.so.2
74e59573d000 r-xp 00002000 103:05 790321 168 168 17 16 168 28 0 0 0 0 0 0 0 0 0 0 ld-linux-x86-64.so.2
74e595767000 r--p 0002c000 103:05 790321 44 44 0 0 44 0 0 0 0 0 0 0 0 0 0 0 ld-linux-x86-64.so.2
74e595773000 r--p 00037000 103:05 790321 8 8 4 4 8 8 0 0 0 0 0 0 0 0 0 0 ld-linux-x86-64.so.2
74e595775000 rw-p 00039000 103:05 790321 8 8 6 6 8 8 0 0 0 0 0 0 0 0 0 0 ld-linux-x86-64.so.2
7ffce80ba000 rw-p 00000000 00:00 0 140 140 77 77 140 140 0 0 0 0 0 0 0 0 0 0
7ffce8185000 r-xp 00000000 00:00 0 8 8 2 2 8 8 0 0 0 0 0 0 0 0 0 0
ffffffffff600000 --xp 00000000 00:00 0 4 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 [vsyscall]
========== ====== ====== ========= ========== ========= === ======== ============== ============= ============== =============== ==== ======= ====== ===========
2810104976 117428 104910 11958 117428 18324 0 0 0 0 0 0 0 0 0 0 KB
for this dump:
[FATAL src/AutoRemoteSyscalls.cc:905:check_syscall_result()]
(task 91468 (rec:89587) at time 1751)
-> Assertion `false' failed to hold. Syscall mprotect failed with errno ENOMEM arg1=0x7399d8001000 arg2=0x14afffff000 arg3=0x2 arg4=0x32 arg5=0xffffffffffffffff arg6=0x0
Tail of trace dump:
{
real_time:21023.575407 global_time:1731, event:`SYSCALL: ioctl' (state:ENTERING_SYSCALL) tid:89587, ticks:11795527
rax:0xffffffffffffffda rbx:0x681fffa0 rcx:0xffffffffffffffff rdx:0x7ffce80d9db0 rsi:0x5401 rdi:0x5 rbp:0x60a593369800 rsp:0x681ffdd0 r8:0x7ffce80d9df0 r9:0x1 r10:0x1000 r11:0x246 r12:0x16 r13:0x7ffce80da000 r14:0x7ffce80da000 r15:0x74e5955aea80 rip:0x70000002 eflags:0x246 cs:0x33 ss:0x2b ds:0x0 es:0x0 fs:0x0 gs:0x0 orig_rax:0x10 fs_base:0x74e5955aea80 gs_base:0x0
}
{
real_time:21023.575412 global_time:1732, event:`SYSCALLBUF_RESET' tid:89587, ticks:11795527
}
{
real_time:21023.575440 global_time:1733, event:`SYSCALL: ioctl' (state:EXITING_SYSCALL) tid:89587, ticks:11795527
rax:0xffffffffffffffea rbx:0x681fffa0 rcx:0xffffffffffffffff rdx:0x7ffce80d9db0 rsi:0x5401 rdi:0x5 rbp:0x60a593369800 rsp:0x681ffdd0 r8:0x7ffce80d9df0 r9:0x1 r10:0x1000 r11:0x246 r12:0x16 r13:0x7ffce80da000 r14:0x7ffce80da000 r15:0x74e5955aea80 rip:0x70000002 eflags:0x246 cs:0x33 ss:0x2b ds:0x0 es:0x0 fs:0x0 gs:0x0 orig_rax:0x10 fs_base:0x74e5955aea80 gs_base:0x0
{ tid:89587, addr:0x7ffce80d9db0, length:0x24 }
}
{
real_time:21023.575684 global_time:1734, event:`SYSCALLBUF_FLUSH' tid:89587, ticks:11824330
{ syscall:'read', ret:0x1000, size:0x1010, desched:1 }
{ syscall:'close', ret:0x0, size:0x10 }
{ syscall:'mprotect', ret:0x0, size:0x10 }
{ syscall:'mprotect', ret:0x0, size:0x10 }
{ syscall:'mprotect', ret:0x0, size:0x10 }
{ syscall:'mprotect', ret:0x0, size:0x10 }
{ syscall:'mprotect', ret:0x0, size:0x10 }
{ start:0x74e4d8000000, size:10000, prot:'r--' }
{ start:0x74e4d8010000, size:4000, prot:'rw-' }
{ start:0x74e4d8020000, size:4000, prot:'rw-' }
{ start:0x74e4d8030000, size:4000, prot:'rw-' }
{ start:0x74e4d8040000, size:4000, prot:'rw-' }
}
{
real_time:21023.575690 global_time:1735, event:`SYSCALL: brk' (state:ENTERING_SYSCALL) tid:89587, ticks:11824330
rax:0xffffffffffffffda rbx:0x60a593413000 rcx:0xffffffffffffffff rdx:0x0 rsi:0x74e588c1ac80 rdi:0x60a593413000 rbp:0x74e588c22218 rsp:0x7ffce80d9d68 r8:0x0 r9:0x7f r10:0x60a5933f1000 r11:0x246 r12:0x60a5933f1000 r13:0x60a5933e19e0 r14:0x1000 r15:0x114b0 rip:0x74e588b1a82b eflags:0x246 cs:0x33 ss:0x2b ds:0x0 es:0x0 fs:0x0 gs:0x0 orig_rax:0xc fs_base:0x74e5955aea80 gs_base:0x0
}
{
real_time:21023.575695 global_time:1736, event:`SYSCALLBUF_RESET' tid:89587, ticks:11824330
}
{
real_time:21023.575777 global_time:1737, event:`SYSCALL: brk' (state:EXITING_SYSCALL) tid:89587, ticks:11824330
rax:0x60a593413000 rbx:0x60a593413000 rcx:0xffffffffffffffff rdx:0x0 rsi:0x74e588c1ac80 rdi:0x60a593413000 rbp:0x74e588c22218 rsp:0x7ffce80d9d68 r8:0x0 r9:0x7f r10:0x60a5933f1000 r11:0x246 r12:0x60a5933f1000 r13:0x60a5933e19e0 r14:0x1000 r15:0x114b0 rip:0x74e588b1a82b eflags:0x246 cs:0x33 ss:0x2b ds:0x0 es:0x0 fs:0x0 gs:0x0 orig_rax:0xc fs_base:0x74e5955aea80 gs_base:0x0
{ map_file:"<ZERO>", addr:0x60a5933f1000, length:0x22000, prot_flags:"rw-p", file_offset:0x0, device:0, inode:0, data_file:"", data_offset:0x0, file_size:0x22000 }
}
{
real_time:21023.669123 global_time:1738, event:`SYSCALLBUF_FLUSH' tid:89587, ticks:21430115
{ syscall:'mprotect', ret:0x0, size:0x10 }
{ syscall:'mprotect', ret:0x0, size:0x10 }
{ syscall:'mprotect', ret:0x0, size:0x10 }
{ syscall:'mprotect', ret:0x0, size:0x10 }
{ syscall:'clock_gettime', ret:0x0, size:0x20 }
{ syscall:'clock_gettime', ret:0x0, size:0x20 }
{ start:0x35c00040000, size:40000, prot:'rw-' }
{ start:0x282a00100000, size:40000, prot:'rw-' }
{ start:0x74e56e800000, size:10000, prot:'rw-' }
{ start:0x74e56e800000, size:10000, prot:'r--' }
}
{
real_time:21023.669131 global_time:1739, event:`SYSCALL: mmap' (state:ENTERING_SYSCALL) tid:89587, ticks:21430115
rax:0xffffffffffffffda rbx:0x681fffa0 rcx:0xffffffffffffffff rdx:0x0 rsi:0x10000 rdi:0x35c00010000 rbp:0x7ffce80da120 rsp:0x681ffdf0 r8:0xffffffff r9:0x0 r10:0x32 r11:0x246 r12:0x7ffce80db848 r13:0x60a5662a8930 r14:0x0 r15:0x74e595775040 rip:0x70000002 eflags:0x246 cs:0x33 ss:0x2b ds:0x0 es:0x0 fs:0x0 gs:0x0 orig_rax:0x9 fs_base:0x74e5955aea80 gs_base:0x0
}
{
real_time:21023.669136 global_time:1740, event:`SYSCALLBUF_RESET' tid:89587, ticks:21430115
}
{
real_time:21023.669186 global_time:1741, event:`SYSCALL: mmap' (state:EXITING_SYSCALL) tid:89587, ticks:21430115
rax:0x35c00010000 rbx:0x681fffa0 rcx:0xffffffffffffffff rdx:0x0 rsi:0x10000 rdi:0x35c00010000 rbp:0x7ffce80da120 rsp:0x681ffdf0 r8:0xffffffff r9:0x0 r10:0x32 r11:0x246 r12:0x7ffce80db848 r13:0x60a5662a8930 r14:0x0 r15:0x74e595775040 rip:0x70000002 eflags:0x246 cs:0x33 ss:0x2b ds:0x0 es:0x0 fs:0x0 gs:0x0 orig_rax:0x9 fs_base:0x74e5955aea80 gs_base:0x0
{ map_file:"<ZERO>", addr:0x35c00010000, length:0x10000, prot_flags:"---p", file_offset:0x0, device:0, inode:0, data_file:"", data_offset:0x0, file_size:0x10000 }
}
{
real_time:21023.669648 global_time:1742, event:`SYSCALLBUF_FLUSH' tid:89587, ticks:21509398
{ syscall:'clock_gettime', ret:0x0, size:0x20 }
{ syscall:'clock_gettime', ret:0x0, size:0x20 }
{ syscall:'openat', ret:0x5, size:0x10, desched:1 }
{ syscall:'readlinkat', ret:0xc, size:0x1c }
{ syscall:'fstatat', ret:0x0, size:0xa0 }
}
{
real_time:21023.669656 global_time:1743, event:`SYSCALL: ioctl' (state:ENTERING_SYSCALL) tid:89587, ticks:21509398
rax:0xffffffffffffffda rbx:0x681fffa0 rcx:0xffffffffffffffff rdx:0x7ffce80d9a30 rsi:0x5401 rdi:0x5 rbp:0x60a593369800 rsp:0x681ffdd0 r8:0x7ffce80d9a70 r9:0x1 r10:0x1000 r11:0x246 r12:0x16 r13:0x7ffce80d9c80 r14:0x7ffce80d9c80 r15:0x74e5955aea80 rip:0x70000002 eflags:0x246 cs:0x33 ss:0x2b ds:0x0 es:0x0 fs:0x0 gs:0x0 orig_rax:0x10 fs_base:0x74e5955aea80 gs_base:0x0
}
{
real_time:21023.669661 global_time:1744, event:`SYSCALLBUF_RESET' tid:89587, ticks:21509398
}
{
real_time:21023.669693 global_time:1745, event:`SYSCALL: ioctl' (state:EXITING_SYSCALL) tid:89587, ticks:21509398
rax:0xffffffffffffffea rbx:0x681fffa0 rcx:0xffffffffffffffff rdx:0x7ffce80d9a30 rsi:0x5401 rdi:0x5 rbp:0x60a593369800 rsp:0x681ffdd0 r8:0x7ffce80d9a70 r9:0x1 r10:0x1000 r11:0x246 r12:0x16 r13:0x7ffce80d9c80 r14:0x7ffce80d9c80 r15:0x74e5955aea80 rip:0x70000002 eflags:0x246 cs:0x33 ss:0x2b ds:0x0 es:0x0 fs:0x0 gs:0x0 orig_rax:0x10 fs_base:0x74e5955aea80 gs_base:0x0
{ tid:89587, addr:0x7ffce80d9a30, length:0x24 }
}
{
real_time:21023.789539 global_time:1746, event:`SYSCALLBUF_FLUSH' tid:89587, ticks:35609795
{ syscall:'read', ret:0x1000, size:0x1010, desched:1 }
{ syscall:'close', ret:0x0, size:0x10 }
{ syscall:'mprotect', ret:0x0, size:0x10 }
{ syscall:'clock_gettime', ret:0x0, size:0x20 }
{ syscall:'clock_gettime', ret:0x0, size:0x20 }
{ syscall:'clock_gettime', ret:0x0, size:0x20 }
{ start:0x35c00180000, size:40000, prot:'rw-' }
}
{
real_time:21023.789550 global_time:1747, event:`SYSCALL: mmap' (state:ENTERING_SYSCALL) tid:89587, ticks:35609795
rax:0xffffffffffffffda rbx:0x681fffa0 rcx:0xffffffffffffffff rdx:0x3 rsi:0x44000 rdi:0x0 rbp:0x43270 rsp:0x681ffdf0 r8:0xffffffff r9:0x0 r10:0x22 r11:0x246 r12:0x0 r13:0x43270 r14:0x1000 r15:0x4325 rip:0x70000002 eflags:0x246 cs:0x33 ss:0x2b ds:0x0 es:0x0 fs:0x0 gs:0x0 orig_rax:0x9 fs_base:0x74e5955aea80 gs_base:0x0
}
{
real_time:21023.789555 global_time:1748, event:`SYSCALLBUF_RESET' tid:89587, ticks:35609795
}
{
real_time:21023.789599 global_time:1749, event:`SYSCALL: mmap' (state:EXITING_SYSCALL) tid:89587, ticks:35609795
rax:0x74e588891000 rbx:0x681fffa0 rcx:0xffffffffffffffff rdx:0x3 rsi:0x44000 rdi:0x0 rbp:0x43270 rsp:0x681ffdf0 r8:0xffffffff r9:0x0 r10:0x22 r11:0x246 r12:0x0 r13:0x43270 r14:0x1000 r15:0x4325 rip:0x70000002 eflags:0x246 cs:0x33 ss:0x2b ds:0x0 es:0x0 fs:0x0 gs:0x0 orig_rax:0x9 fs_base:0x74e5955aea80 gs_base:0x0
{ map_file:"<ZERO>", addr:0x74e588891000, length:0x44000, prot_flags:"rw-p", file_offset:0x0, device:0, inode:0, data_file:"", data_offset:0x0, file_size:0x44000 }
}
{
real_time:21023.789789 global_time:1750, event:`SYSCALL: mmap' (state:ENTERING_SYSCALL) tid:89587, ticks:35644664
rax:0xffffffffffffffda rbx:0x681fffa0 rcx:0xffffffffffffffff rdx:0x0 rsi:0x14afffff000 rdi:0x35200000000 rbp:0x7ffce80da940 rsp:0x681ffdf0 r8:0x4 r9:0x0 r10:0x2 r11:0x246 r12:0x7ffce80db848 r13:0x60a5662a8930 r14:0x0 r15:0x74e595775040 rip:0x70000002 eflags:0x246 cs:0x33 ss:0x2b ds:0x0 es:0x0 fs:0x0 gs:0x0 orig_rax:0x9 fs_base:0x74e5955aea80 gs_base:0x0
}
{
real_time:21023.790585 global_time:1751, event:`SYSCALL: mmap' (state:EXITING_SYSCALL) tid:89587, ticks:35644664
rax:0x7399d8001000 rbx:0x681fffa0 rcx:0xffffffffffffffff rdx:0x0 rsi:0x14afffff000 rdi:0x35200000000 rbp:0x7ffce80da940 rsp:0x681ffdf0 r8:0x4 r9:0x0 r10:0x2 r11:0x246 r12:0x7ffce80db848 r13:0x60a5662a8930 r14:0x0 r15:0x74e595775040 rip:0x70000002 eflags:0x246 cs:0x33 ss:0x2b ds:0x0 es:0x0 fs:0x0 gs:0x0 orig_rax:0x9 fs_base:0x74e5955aea80 gs_base:0x0
{ map_file:"/memfd:V8MemFDForTesting (deleted)", addr:0x7399d8001000, length:0x14afffff000, prot_flags:"---p", file_offset:0x0, device:1, inode:167048, data_file:"", data_offset:0x0, file_size:0x14a00000000 }
{ tid:89587, addr:0x7399d8001000, length:0x14a00000000, holes:[(nil)-0x9000000000x900007000-0x90000f0000x900010000-0x9000200000x90005f000-0x9000800000x900183000-0x9001c00000x9001f4000-0x14a00000000] }
}
{
real_time:21023.790624 global_time:1752, event:`SYSCALL: munmap' (state:ENTERING_SYSCALL) tid:89587, ticks:35644698
rax:0xffffffffffffffda rbx:0x681fffa0 rcx:0xffffffffffffffff rdx:0x74e595694667 rsi:0x27fff000 rdi:0x7399d8001000 rbp:0x7ffce80da940 rsp:0x681ffdf0 r8:0x4 r9:0x0 r10:0x2 r11:0x246 r12:0x7ffce80db848 r13:0x60a5662a8930 r14:0x0 r15:0x74e595775040 rip:0x70000002 eflags:0x246 cs:0x33 ss:0x2b ds:0x0 es:0x0 fs:0x0 gs:0x0 orig_rax:0xb fs_base:0x74e5955aea80 gs_base:0x0
}
=== Start rr backtrace:
rr(_ZN2rr13dump_rr_stackERNS_8ScopedFdE+0x5b)[0x5e1b55d94a0b]
rr(_ZN2rr15emergency_debugEPNS_4TaskE+0x152)[0x5e1b55c67f12]
rr(+0xdfa82)[0x5e1b55c68a82]
rr(+0xe008f)[0x5e1b55c6908f]
rr(_ZN2rr18AutoRemoteSyscalls20check_syscall_resultElib+0x4ae)[0x5e1b55bfa64e]
rr(+0x1db393)[0x5e1b55d64393]
rr(_ZN2rr4Task35write_bytes_helper_no_notificationsENS_10remote_ptrIvEElPKvPbj+0x148)[0x5e1b55d67698]
rr(_ZN2rr4Task18write_bytes_helperENS_10remote_ptrIvEElPKvPbj+0x20)[0x5e1b55d679b0]
rr(+0x18cc3c)[0x5e1b55d15c3c]
rr(_ZN2rr10ReplayTask28apply_data_record_from_traceEv+0xae)[0x5e1b55d15f0e]
rr(+0x1887da)[0x5e1b55d117da]
rr(+0x18ef7f)[0x5e1b55d17f7f]
rr(_ZN2rr13ReplaySession28setup_replay_one_trace_frameEPNS_10ReplayTaskE+0x994)[0x5e1b55d0a8b4]
rr(_ZN2rr13ReplaySession11replay_stepERKNS0_15StepConstraintsE+0xf9)[0x5e1b55d0ab19]
rr(_ZN2rr14ReplayTimeline19replay_step_forwardENS_10RunCommandE+0xbf)[0x5e1b55d27aef]
rr(_ZN2rr9GdbServer14debug_one_stepERNS_10GdbRequestE+0xf48)[0x5e1b55c4c5b8]
rr(_ZN2rr9GdbServer12serve_replayESt10shared_ptrINS_13ReplaySessionEERKNS0_6TargetEPVbNS_12DebuggerTypeERKNS0_15ConnectionFlagsE+0x977)[0x5e1b55c4d3b7]
rr(_ZN2rr13ReplayCommand3runERSt6vectorINSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEESaIS7_EE+0xd18)[0x5e1b55cfd708]
rr(main+0x1a1)[0x5e1b55bde441]
/lib/x86_64-linux-gnu/libc.so.6(+0x29d90)[0x7b83a4029d90]
/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0x80)[0x7b83a4029e40]
rr(_start+0x25)[0x5e1b55be0f25]
=== End rr backtrace
With steps it is hard, for my huge binary it fails every time but I couldn't minimize it. Basically I just create a CoW clone through mmap MAP_PRIVATE using memfd files
@rocallahan This fix broke rr in one of my environments... (RHEL 9.2 with 5.14.0-284.25.1.el9_2.x86_64), or it possibly was always broken and only the new test shows that this is the case.
... but then, it seems the test is possible "too huge" to run on this machine, from top -c:
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
367121 so 20 0 15880 11388 6660 R 92.8 0.0 0:25.83 rr --suppress-environment-warnings --fatal-errors --check-c+
367122 so 20 0 1024.0g 60.3g 60.3g t 1.9 48.1 0:00.47 [rr:mmap_huge-4q]
367121 so 20 0 15880 8816 4608 R 59.2 0.0 1:07.92 rr --suppress-environment-warnings --fatal-errors --check-c+
232 root 20 0 0 0 0 D 53.3 0.0 9:11.65 [kswapd1]
231 root 20 0 0 0 0 D 49.8 0.0 9:12.75 [kswapd0]
Should we really run an automated test (which may also be run in parallel to the no-syscallbuf version so takes double size) that takes 1TB of virtual ram and > 58GB of memory?
$> ctest -VV -R mmap_huge
UpdateCTestConfiguration from :/tmp/rr_build/DartConfiguration.tcl
UpdateCTestConfiguration from :/tmp/rr_build/DartConfiguration.tcl
Test project /tmp/rr_build
Constructing a list of tests
Done constructing a list of tests
Updating test list for fixtures
Added 0 tests to meet fixture requirements
Checking test dependency graph...
Checking test dependency graph end
test 379
Start 379: mmap_huge
379: Test command: /usr/bin/bash "source_dir/src/test/basic_test.run" "mmap_huge" "" "bin_dir" "120"
379: Test timeout computed to be: 10000000
379: source_dir/src/test/util.sh: line 324: 366545 Aborted (core dumped) _RR_TRACE_DIR="$workdir" test-monitor $TIMEOUT replay.err $RR_EXE $GLOBAL_OPTIONS replay --retry-transient-errors -a $replayflags > replay.out 2> replay.err
379: Test 'mmap_huge' FAILED: : error during replay:
379: --------------------------------------------------
379: timeout 120 exceeded
379: ====== /proc/366546/status
379: Name: rr
379: Umask: 0002
379: State: D (disk sleep)
379: Tgid: 366546
379: Ngid: 0
379: Pid: 366546
379: PPid: 366545
379: TracerPid: 0
379: Uid: 12427 12427 12427 12427
379: Gid: 613 613 613 613
379: FDSize: 64
379: Groups: 613 617 12430
379: NStgid: 366546
379: NSpid: 366546
379: NSpgid: 366546
379: NSsid: 360106
379: VmPeak: 16928 kB
379: VmSize: 16928 kB
379: VmLck: 0 kB
379: VmPin: 0 kB
379: VmHWM: 11476 kB
379: VmRSS: 6612 kB
379: RssAnon: 4204 kB
379: RssFile: 2408 kB
379: RssShmem: 0 kB
379: VmData: 4792 kB
379: VmStk: 136 kB
379: VmExe: 2356 kB
379: VmLib: 4676 kB
379: VmPTE: 72 kB
379: VmSwap: 548 kB
379: HugetlbPages: 0 kB
379: CoreDumping: 0
379: THP_enabled: 1
379: Threads: 1
379: SigQ: 0/513428
379: SigPnd: 0000000000000000
379: ShdPnd: 0000000000000000
379: SigBlk: 0000000000000000
379: SigIgn: 0000000000000000
379: SigCgt: 0000000000002000
379: CapInh: 0000000000000000
379: CapPrm: 0000000000000000
379: CapEff: 0000000000000000
379: CapBnd: 000001ffffffffff
379: CapAmb: 0000000000000000
379: NoNewPrivs: 0
379: Seccomp: 0
379: Seccomp_filters: 0
379: Speculation_Store_Bypass: thread vulnerable
379: SpeculationIndirectBranch: conditional enabled
379: Cpus_allowed: 00000100
379: Cpus_allowed_list: 8
379: Mems_allowed: 00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000003
379: Mems_allowed_list: 0-1
379: voluntary_ctxt_switches: 9035
379: nonvoluntary_ctxt_switches: 124420
379: ====== /proc/366546/stack
379: ====== /proc/366547/status
379: Name: rr:mmap_huge-hC
379: Umask: 0002
379: State: t (tracing stop)
379: Tgid: 366547
379: Ngid: 0
379: Pid: 366547
379: PPid: 366546
379: TracerPid: 366546
379: Uid: 12427 12427 12427 12427
379: Gid: 613 613 613 613
379: FDSize: 1024
379: Groups: 613 617 12430
379: NStgid: 366547
379: NSpid: 366547
379: NSpgid: 366547
379: NSsid: 366547
379: VmPeak: 1073747528 kB
379: VmSize: 1073747528 kB
379: VmLck: 0 kB
379: VmPin: 0 kB
379: VmHWM: 127865252 kB
379: VmRSS: 126871560 kB
379: RssAnon: 4 kB
379: RssFile: 0 kB
379: RssShmem: 126871556 kB
379: VmData: 2328 kB
379: VmStk: 0 kB
379: VmExe: 4 kB
379: VmLib: 1708 kB
379: VmPTE: 288152 kB
379: VmSwap: 2364 kB
379: HugetlbPages: 0 kB
379: CoreDumping: 0
379: THP_enabled: 1
379: Threads: 1
379: SigQ: 0/513428
379: SigPnd: 0000000000000000
379: ShdPnd: 0000000000000000
379: SigBlk: 0000000000000000
379: SigIgn: 0000000000010000
379: SigCgt: 0000000000000000
379: CapInh: 0000000000000000
379: CapPrm: 0000000000000000
379: CapEff: 0000000000000000
379: CapBnd: 000001ffffffffff
379: CapAmb: 0000000000000000
379: NoNewPrivs: 1
379: Seccomp: 0
379: Seccomp_filters: 0
379: Speculation_Store_Bypass: thread vulnerable
379: SpeculationIndirectBranch: conditional enabled
379: Cpus_allowed: 00000100
379: Cpus_allowed_list: 8
379: Mems_allowed: 00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000003
379: Mems_allowed_list: 0-1
379: voluntary_ctxt_switches: 108386
379: nonvoluntary_ctxt_switches: 38
379: ====== /proc/366547/stack
379: ====== gdb -p 366546 -ex 'set confirm off' -ex 'set height 0' -ex 'thread apply all bt' -ex q </dev/null 2>&1
379: GNU gdb (GDB) 13.2
379: Copyright (C) 2023 Free Software Foundation, Inc.
379: License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
379: This is free software: you are free to change and redistribute it.
379: There is NO WARRANTY, to the extent permitted by law.
379: Type "show copying" and "show warranty" for details.
379: This GDB was configured as "x86_64-pc-linux-gnu".
379: Type "show configuration" for configuration details.
379: For bug reporting instructions, please see:
379: <https://www.gnu.org/software/gdb/bugs/>.
379: Find the GDB manual and other documentation resources online at:
379: <http://www.gnu.org/software/gdb/documentation/>.
379:
379: For help, type "help".
379: Type "apropos word" to search for commands related to "word".
379: Attaching to process 366546
379: Reading symbols from /tmp/rr_build/bin/rr...
379: Reading symbols from /usr/lib64/libz.so.1...
379: Reading symbols from .gnu_debugdata for /usr/lib64/libz.so.1...
379: (No debugging symbols found in .gnu_debugdata for /usr/lib64/libz.so.1)
379: Reading symbols from /usr/lib64/libzstd.so.1...
379: Reading symbols from .gnu_debugdata for /usr/lib64/libzstd.so.1...
379: (No debugging symbols found in .gnu_debugdata for /usr/lib64/libzstd.so.1)
379: Reading symbols from /usr/lib64/libcapnp.so.0.10.3...
379: Reading symbols from .gnu_debugdata for /usr/lib64/libcapnp.so.0.10.3...
379: (No debugging symbols found in .gnu_debugdata for /usr/lib64/libcapnp.so.0.10.3)
379: Reading symbols from /usr/lib64/libkj.so.0.10.3...
379: Reading symbols from .gnu_debugdata for /usr/lib64/libkj.so.0.10.3...
379: (No debugging symbols found in .gnu_debugdata for /usr/lib64/libkj.so.0.10.3)
379: Reading symbols from /usr/lib64/libstdc++.so.6...
379: Reading symbols from .gnu_debugdata for /usr/lib64/libstdc++.so.6...
379: (No debugging symbols found in .gnu_debugdata for /usr/lib64/libstdc++.so.6)
379: Reading symbols from /usr/lib64/libm.so.6...
379: Reading symbols from .gnu_debugdata for /usr/lib64/libm.so.6...
379: (No debugging symbols found in .gnu_debugdata for /usr/lib64/libm.so.6)
379: Reading symbols from /usr/lib64/libgcc_s.so.1...
379: Reading symbols from .gnu_debugdata for /usr/lib64/libgcc_s.so.1...
379: (No debugging symbols found in .gnu_debugdata for /usr/lib64/libgcc_s.so.1)
379: Reading symbols from /usr/lib64/libc.so.6...
379: (No debugging symbols found in /usr/lib64/libc.so.6)
379: Reading symbols from /lib64/ld-linux-x86-64.so.2...
379: [Thread debugging using libthread_db enabled]
379: Using host libthread_db library "/usr/lib64/libthread_db.so.1".
379: 0x00007f7bc1f3c94a in pwrite64 () from /usr/lib64/libc.so.6
379:
379: Thread 1 (Thread 0x7f7bc2486740 (LWP 366546) "rr"):
379: #0 0x00007f7bc1f3c94a in pwrite64 () from /usr/lib64/libc.so.6
379: #1 0x00000000005bdccb in rr::pwrite_all_fallible (offset=138917883940864, size=4194304, buf=0x7f7bc19ff010, fd=13) at /tmp/rr/src/util.cc:2229
379: #2 rr::safe_pwrite64 (t=t@entry=0x20d1de0, buf=buf@entry=0x7f7bc19ff010, buf_size=buf_size@entry=4194304, addr=addr@entry=...) at /tmp/rr/src/Task.cc:3243
379: #3 0x00000000005c0726 in rr::Task::write_bytes_helper_no_notifications (this=this@entry=0x20d1de0, addr=addr@entry=..., buf_size=buf_size@entry=4194304, buf=0x7f7bc19ff010, ok=ok@entry=0x0, flags=flags@entry=0) at /tmp/rr/src/preload/../remote_ptr.h:28
379: #4 0x00000000005c0f07 in rr::Task::write_zeroes (this=0x20d1de0, remote=0x7fff784d5148, addr=..., size=947853979648) at /usr/include/c++/11/bits/stl_vector.h:1168
379: #5 0x00000000005740d1 in rr::write_data_with_holes (t=t@entry=0x20d1de0, buf=...) at /tmp/rr/src/ReplayTask.cc:151
379: #6 0x00000000005741ca in rr::ReplayTask::apply_data_record_from_trace (this=0x20d1de0) at /tmp/rr/src/ReplayTask.cc:182
379: #7 0x000000000056e152 in rr::finish_shared_mmap (t=t@entry=0x20d1de0, remote=..., rec_addr=..., rec_addr@entry=..., length=length@entry=1099511627776, prot=prot@entry=0, flags=flags@entry=1, fds=..., offset_bytes=0, km=..., data=...) at /tmp/rr/src/replay_syscall.cc:656
379: #8 0x000000000056f85c in rr::process_mmap (t=0x20d1de0, trace_frame=..., length=1099511627776, prot=0, flags=1, fd=4, offset_bytes=0, step=0x20cf0e8) at /tmp/rr/src/replay_syscall.cc:746
379: #9 0x000000000057604f in rr::rep_process_syscall_arch<rr::X64Arch> (t=0x20d1de0, step=0x20cf0e8, trace_regs=...) at /tmp/rr/src/replay_syscall.cc:1180
379: #10 0x0000000000569a83 in rr::rep_process_syscall (step=0x20cf0e8, t=<optimized out>) at /tmp/rr/src/replay_syscall.cc:1410
379: #11 rr::ReplaySession::setup_replay_one_trace_frame (this=this@entry=0x20ceb10, t=t@entry=0x20d1de0) at /tmp/rr/src/ReplaySession.cc:1976
379: #12 0x0000000000569c99 in rr::ReplaySession::replay_step (this=0x20ceb10, constraints=...) at /tmp/rr/src/ReplaySession.cc:2040
379: #13 0x000000000055e322 in rr::ReplaySession::replay_step (command=<optimized out>, this=<optimized out>) at /tmp/rr/src/ReplaySession.h:294
379: #14 rr::serve_replay_no_debugger (flags=..., trace_dir=...) at /tmp/rr/src/ReplayCommand.cc:402
379: #15 rr::replay (flags=..., trace_dir=...) at /tmp/rr/src/ReplayCommand.cc:512
379: #16 rr::ReplayCommand::run (this=<optimized out>, args=...) at /tmp/rr/src/ReplayCommand.cc:695
379: #17 0x000000000044e9d1 in main (argc=<optimized out>, argv=<optimized out>) at /tmp/rr/src/main.cc:278
379: Detaching from program: /tmp/rr_build/bin/rr, process 366546
379: [Inferior 1 (process 366546) detached]
379: ====== gdb -p 366546 -ex 'set confirm off' -ex 'set height 0' -ex 'b rr::force_close_record_session' -ex 'p rr::force_close_record_session()' -ex detach -ex q </dev/null 2>&1
379: GNU gdb (GDB) 13.2
379: Copyright (C) 2023 Free Software Foundation, Inc.
379: License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
379: This is free software: you are free to change and redistribute it.
379: There is NO WARRANTY, to the extent permitted by law.
379: Type "show copying" and "show warranty" for details.
379: This GDB was configured as "x86_64-pc-linux-gnu".
379: Type "show configuration" for configuration details.
379: For bug reporting instructions, please see:
379: <https://www.gnu.org/software/gdb/bugs/>.
379: Find the GDB manual and other documentation resources online at:
379: <http://www.gnu.org/software/gdb/documentation/>.
379:
379: For help, type "help".
379: Type "apropos word" to search for commands related to "word".
379: Attaching to process 366546
379: Reading symbols from /tmp/rr_build/bin/rr...
379: Reading symbols from /usr/lib64/libz.so.1...
379: Reading symbols from .gnu_debugdata for /usr/lib64/libz.so.1...
379: (No debugging symbols found in .gnu_debugdata for /usr/lib64/libz.so.1)
379: Reading symbols from /usr/lib64/libzstd.so.1...
379: Reading symbols from .gnu_debugdata for /usr/lib64/libzstd.so.1...
379: (No debugging symbols found in .gnu_debugdata for /usr/lib64/libzstd.so.1)
379: Reading symbols from /usr/lib64/libcapnp.so.0.10.3...
379: Reading symbols from .gnu_debugdata for /usr/lib64/libcapnp.so.0.10.3...
379: (No debugging symbols found in .gnu_debugdata for /usr/lib64/libcapnp.so.0.10.3)
379: Reading symbols from /usr/lib64/libkj.so.0.10.3...
379: Reading symbols from .gnu_debugdata for /usr/lib64/libkj.so.0.10.3...
379: (No debugging symbols found in .gnu_debugdata for /usr/lib64/libkj.so.0.10.3)
379: Reading symbols from /usr/lib64/libstdc++.so.6...
379: Reading symbols from .gnu_debugdata for /usr/lib64/libstdc++.so.6...
379: (No debugging symbols found in .gnu_debugdata for /usr/lib64/libstdc++.so.6)
379: Reading symbols from /usr/lib64/libm.so.6...
379: Reading symbols from .gnu_debugdata for /usr/lib64/libm.so.6...
379: (No debugging symbols found in .gnu_debugdata for /usr/lib64/libm.so.6)
379: Reading symbols from /usr/lib64/libgcc_s.so.1...
379: Reading symbols from .gnu_debugdata for /usr/lib64/libgcc_s.so.1...
379: (No debugging symbols found in .gnu_debugdata for /usr/lib64/libgcc_s.so.1)
379: Reading symbols from /usr/lib64/libc.so.6...
379: (No debugging symbols found in /usr/lib64/libc.so.6)
379: Reading symbols from /lib64/ld-linux-x86-64.so.2...
379: [Thread debugging using libthread_db enabled]
379: Using host libthread_db library "/usr/lib64/libthread_db.so.1".
379: 0x00007f7bc1f3c94a in pwrite64 () from /usr/lib64/libc.so.6
379: Breakpoint 1 at 0x4f3f40: file /tmp/rr/src/RecordCommand.cc, line 635.
379:
379: Breakpoint 1, rr::force_close_record_session ()
379: at /tmp/rr/src/RecordCommand.cc:635
379: 635 if (static_session) {
379: The program being debugged stopped while in a function called from GDB.
379: Evaluation of the expression containing the function
379: (rr::force_close_record_session()) will be abandoned.
379: When the function is done executing, GDB will silently stop.
379: Detaching from program: /tmp/rr_build/bin/rr, process 366546
379: [Inferior 1 (process 366546) detached]
379: --------------------------------------------------
379: replay.out:
379: --------------------------------------------------
379: --------------------------------------------------
379: Test mmap_huge failed, leaving behind /tmp/rr-test-mmap_huge-hC5RlRInx
379: To replay the failed test, run
379: _RR_TRACE_DIR=/tmp/rr-test-mmap_huge-hC5RlRInx rr replay
1/2 Test #379: mmap_huge ........................***Failed Error regular expression found in output. Regex=[FAILED]133.80 sec
test 380
Start 380: mmap_huge-no-syscallbuf
380: Test command: /usr/bin/bash "source_dir/src/test/basic_test.run" "mmap_huge" "-n" "bin_dir" "120"
380: Test timeout computed to be: 10000000
380: source_dir/src/test/util.sh: line 324: 367120 Aborted (core dumped) _RR_TRACE_DIR="$workdir" test-monitor $TIMEOUT replay.err $RR_EXE $GLOBAL_OPTIONS replay --retry-transient-errors -a $replayflags > replay.out 2> replay.err
380: Test 'mmap_huge' FAILED: : error during replay:
380: --------------------------------------------------
380: timeout 120 exceeded
380: ====== /proc/367121/status
380: Name: rr
380: Umask: 0002
380: State: D (disk sleep)
380: Tgid: 367121
380: Ngid: 0
380: Pid: 367121
380: PPid: 367120
380: TracerPid: 0
380: Uid: 12427 12427 12427 12427
380: Gid: 613 613 613 613
380: FDSize: 64
380: Groups: 613 617 12430
380: NStgid: 367121
380: NSpid: 367121
380: NSpgid: 367121
380: NSsid: 360106
380: VmPeak: 15880 kB
380: VmSize: 15880 kB
380: VmLck: 0 kB
380: VmPin: 0 kB
380: VmHWM: 11388 kB
380: VmRSS: 6748 kB
380: RssAnon: 4208 kB
380: RssFile: 2540 kB
380: RssShmem: 0 kB
380: VmData: 4768 kB
380: VmStk: 136 kB
380: VmExe: 2356 kB
380: VmLib: 4676 kB
380: VmPTE: 68 kB
380: VmSwap: 520 kB
380: HugetlbPages: 0 kB
380: CoreDumping: 0
380: THP_enabled: 1
380: Threads: 1
380: SigQ: 0/513428
380: SigPnd: 0000000000000000
380: ShdPnd: 0000000000000000
380: SigBlk: 0000000000000000
380: SigIgn: 0000000000000000
380: SigCgt: 0000000000002000
380: CapInh: 0000000000000000
380: CapPrm: 0000000000000000
380: CapEff: 0000000000000000
380: CapBnd: 000001ffffffffff
380: CapAmb: 0000000000000000
380: NoNewPrivs: 0
380: Seccomp: 0
380: Seccomp_filters: 0
380: Speculation_Store_Bypass: thread vulnerable
380: SpeculationIndirectBranch: conditional enabled
380: Cpus_allowed: 00000080
380: Cpus_allowed_list: 7
380: Mems_allowed: 00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000003
380: Mems_allowed_list: 0-1
380: voluntary_ctxt_switches: 7972
380: nonvoluntary_ctxt_switches: 130121
380: ====== /proc/367121/stack
380: ====== /proc/367122/status
380: Name: rr:mmap_huge-4q
380: Umask: 0002
380: State: t (tracing stop)
380: Tgid: 367122
380: Ngid: 0
380: Pid: 367122
380: PPid: 367121
380: TracerPid: 367121
380: Uid: 12427 12427 12427 12427
380: Gid: 613 613 613 613
380: FDSize: 1024
380: Groups: 613 617 12430
380: NStgid: 367122
380: NSpid: 367122
380: NSpgid: 367122
380: NSsid: 367122
380: VmPeak: 1073746500 kB
380: VmSize: 1073746500 kB
380: VmLck: 0 kB
380: VmPin: 0 kB
380: VmHWM: 127996524 kB
380: VmRSS: 126641296 kB
380: RssAnon: 0 kB
380: RssFile: 4 kB
380: RssShmem: 126641292 kB
380: VmData: 2328 kB
380: VmStk: 0 kB
380: VmExe: 4 kB
380: VmLib: 1704 kB
380: VmPTE: 305056 kB
380: VmSwap: 296 kB
380: HugetlbPages: 0 kB
380: CoreDumping: 0
380: THP_enabled: 1
380: Threads: 1
380: SigQ: 0/513428
380: SigPnd: 0000000000000000
380: ShdPnd: 0000000000000000
380: SigBlk: 0000000000000000
380: SigIgn: 0000000000010000
380: SigCgt: 0000000000000000
380: CapInh: 0000000000000000
380: CapPrm: 0000000000000000
380: CapEff: 0000000000000000
380: CapBnd: 000001ffffffffff
380: CapAmb: 0000000000000000
380: NoNewPrivs: 1
380: Seccomp: 0
380: Seccomp_filters: 0
380: Speculation_Store_Bypass: thread vulnerable
380: SpeculationIndirectBranch: conditional enabled
380: Cpus_allowed: 00000080
380: Cpus_allowed_list: 7
380: Mems_allowed: 00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000003
380: Mems_allowed_list: 0-1
380: voluntary_ctxt_switches: 114825
380: nonvoluntary_ctxt_switches: 69
380: ====== /proc/367122/stack
380: ====== gdb -p 367121 -ex 'set confirm off' -ex 'set height 0' -ex 'thread apply all bt' -ex q </dev/null 2>&1
380: GNU gdb (GDB) 13.2
380: Copyright (C) 2023 Free Software Foundation, Inc.
380: License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
380: This is free software: you are free to change and redistribute it.
380: There is NO WARRANTY, to the extent permitted by law.
380: Type "show copying" and "show warranty" for details.
380: This GDB was configured as "x86_64-pc-linux-gnu".
380: Type "show configuration" for configuration details.
380: For bug reporting instructions, please see:
380: <https://www.gnu.org/software/gdb/bugs/>.
380: Find the GDB manual and other documentation resources online at:
380: <http://www.gnu.org/software/gdb/documentation/>.
380:
380: For help, type "help".
380: Type "apropos word" to search for commands related to "word".
380: Attaching to process 367121
380: Reading symbols from /tmp/rr_build/bin/rr...
380: Reading symbols from /usr/lib64/libz.so.1...
380: Reading symbols from .gnu_debugdata for /usr/lib64/libz.so.1...
380: (No debugging symbols found in .gnu_debugdata for /usr/lib64/libz.so.1)
380: Reading symbols from /usr/lib64/libzstd.so.1...
380: Reading symbols from .gnu_debugdata for /usr/lib64/libzstd.so.1...
380: (No debugging symbols found in .gnu_debugdata for /usr/lib64/libzstd.so.1)
380: Reading symbols from /usr/lib64/libcapnp.so.0.10.3...
380: Reading symbols from .gnu_debugdata for /usr/lib64/libcapnp.so.0.10.3...
380: (No debugging symbols found in .gnu_debugdata for /usr/lib64/libcapnp.so.0.10.3)
380: Reading symbols from /usr/lib64/libkj.so.0.10.3...
380: Reading symbols from .gnu_debugdata for /usr/lib64/libkj.so.0.10.3...
380: (No debugging symbols found in .gnu_debugdata for /usr/lib64/libkj.so.0.10.3)
380: Reading symbols from /usr/lib64/libstdc++.so.6...
380: Reading symbols from .gnu_debugdata for /usr/lib64/libstdc++.so.6...
380: (No debugging symbols found in .gnu_debugdata for /usr/lib64/libstdc++.so.6)
380: Reading symbols from /usr/lib64/libm.so.6...
380: Reading symbols from .gnu_debugdata for /usr/lib64/libm.so.6...
380: (No debugging symbols found in .gnu_debugdata for /usr/lib64/libm.so.6)
380: Reading symbols from /usr/lib64/libgcc_s.so.1...
380: Reading symbols from .gnu_debugdata for /usr/lib64/libgcc_s.so.1...
380: (No debugging symbols found in .gnu_debugdata for /usr/lib64/libgcc_s.so.1)
380: Reading symbols from /usr/lib64/libc.so.6...
380: (No debugging symbols found in /usr/lib64/libc.so.6)
380: Reading symbols from /lib64/ld-linux-x86-64.so.2...
380: [Thread debugging using libthread_db enabled]
380: Using host libthread_db library "/usr/lib64/libthread_db.so.1".
380: 0x00007ff85833c94a in pwrite64 () from /usr/lib64/libc.so.6
380:
380: Thread 1 (Thread 0x7ff858891740 (LWP 367121) "rr"):
380: #0 0x00007ff85833c94a in pwrite64 () from /usr/lib64/libc.so.6
380: #1 0x00000000005bdccb in rr::pwrite_all_fallible (offset=139052156194816, size=4194304, buf=0x7ff857dff010, fd=13) at /tmp/rr/src/util.cc:2229
380: #2 rr::safe_pwrite64 (t=t@entry=0x13b6bc0, buf=buf@entry=0x7ff857dff010, buf_size=buf_size@entry=4194304, addr=addr@entry=...) at /tmp/rr/src/Task.cc:3243
380: #3 0x00000000005c0726 in rr::Task::write_bytes_helper_no_notifications (this=this@entry=0x13b6bc0, addr=addr@entry=..., buf_size=buf_size@entry=4194304, buf=0x7ff857dff010, ok=ok@entry=0x0, flags=flags@entry=0) at /tmp/rr/src/preload/../remote_ptr.h:28
380: #4 0x00000000005c0f07 in rr::Task::write_zeroes (this=0x13b6bc0, remote=0x7fff6b1e3aa8, addr=..., size=937225613312) at /usr/include/c++/11/bits/stl_vector.h:1168
380: #5 0x00000000005740d1 in rr::write_data_with_holes (t=t@entry=0x13b6bc0, buf=...) at /tmp/rr/src/ReplayTask.cc:151
380: #6 0x00000000005741ca in rr::ReplayTask::apply_data_record_from_trace (this=0x13b6bc0) at /tmp/rr/src/ReplayTask.cc:182
380: #7 0x000000000056e152 in rr::finish_shared_mmap (t=t@entry=0x13b6bc0, remote=..., rec_addr=..., rec_addr@entry=..., length=length@entry=1099511627776, prot=prot@entry=0, flags=flags@entry=1, fds=..., offset_bytes=0, km=..., data=...) at /tmp/rr/src/replay_syscall.cc:656
380: #8 0x000000000056f85c in rr::process_mmap (t=0x13b6bc0, trace_frame=..., length=1099511627776, prot=0, flags=1, fd=4, offset_bytes=0, step=0x13b40e8) at /tmp/rr/src/replay_syscall.cc:746
380: #9 0x000000000057604f in rr::rep_process_syscall_arch<rr::X64Arch> (t=0x13b6bc0, step=0x13b40e8, trace_regs=...) at /tmp/rr/src/replay_syscall.cc:1180
380: #10 0x0000000000569a83 in rr::rep_process_syscall (step=0x13b40e8, t=<optimized out>) at /tmp/rr/src/replay_syscall.cc:1410
380: #11 rr::ReplaySession::setup_replay_one_trace_frame (this=this@entry=0x13b3b10, t=t@entry=0x13b6bc0) at /tmp/rr/src/ReplaySession.cc:1976
380: #12 0x0000000000569c99 in rr::ReplaySession::replay_step (this=0x13b3b10, constraints=...) at /tmp/rr/src/ReplaySession.cc:2040
380: #13 0x000000000055e322 in rr::ReplaySession::replay_step (command=<optimized out>, this=<optimized out>) at /tmp/rr/src/ReplaySession.h:294
380: #14 rr::serve_replay_no_debugger (flags=..., trace_dir=...) at /tmp/rr/src/ReplayCommand.cc:402
380: #15 rr::replay (flags=..., trace_dir=...) at /tmp/rr/src/ReplayCommand.cc:512
380: #16 rr::ReplayCommand::run (this=<optimized out>, args=...) at /tmp/rr/src/ReplayCommand.cc:695
380: #17 0x000000000044e9d1 in main (argc=<optimized out>, argv=<optimized out>) at /tmp/rr/src/main.cc:278
380: Detaching from program: /tmp/rr_build/bin/rr, process 367121
380: [Inferior 1 (process 367121) detached]
380: ====== gdb -p 367121 -ex 'set confirm off' -ex 'set height 0' -ex 'b rr::force_close_record_session' -ex 'p rr::force_close_record_session()' -ex detach -ex q </dev/null 2>&1
380: GNU gdb (GDB) 13.2
380: Copyright (C) 2023 Free Software Foundation, Inc.
380: License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
380: This is free software: you are free to change and redistribute it.
380: There is NO WARRANTY, to the extent permitted by law.
380: Type "show copying" and "show warranty" for details.
380: This GDB was configured as "x86_64-pc-linux-gnu".
380: Type "show configuration" for configuration details.
380: For bug reporting instructions, please see:
380: <https://www.gnu.org/software/gdb/bugs/>.
380: Find the GDB manual and other documentation resources online at:
380: <http://www.gnu.org/software/gdb/documentation/>.
380:
380: For help, type "help".
380: Type "apropos word" to search for commands related to "word".
380: Attaching to process 367121
380: Reading symbols from /tmp/rr_build/bin/rr...
380: Reading symbols from /usr/lib64/libz.so.1...
380: Reading symbols from .gnu_debugdata for /usr/lib64/libz.so.1...
380: (No debugging symbols found in .gnu_debugdata for /usr/lib64/libz.so.1)
380: Reading symbols from /usr/lib64/libzstd.so.1...
380: Reading symbols from .gnu_debugdata for /usr/lib64/libzstd.so.1...
380: (No debugging symbols found in .gnu_debugdata for /usr/lib64/libzstd.so.1)
380: Reading symbols from /usr/lib64/libcapnp.so.0.10.3...
380: Reading symbols from .gnu_debugdata for /usr/lib64/libcapnp.so.0.10.3...
380: (No debugging symbols found in .gnu_debugdata for /usr/lib64/libcapnp.so.0.10.3)
380: Reading symbols from /usr/lib64/libkj.so.0.10.3...
380: Reading symbols from .gnu_debugdata for /usr/lib64/libkj.so.0.10.3...
380: (No debugging symbols found in .gnu_debugdata for /usr/lib64/libkj.so.0.10.3)
380: Reading symbols from /usr/lib64/libstdc++.so.6...
380: Reading symbols from .gnu_debugdata for /usr/lib64/libstdc++.so.6...
380: (No debugging symbols found in .gnu_debugdata for /usr/lib64/libstdc++.so.6)
380: Reading symbols from /usr/lib64/libm.so.6...
380: Reading symbols from .gnu_debugdata for /usr/lib64/libm.so.6...
380: (No debugging symbols found in .gnu_debugdata for /usr/lib64/libm.so.6)
380: Reading symbols from /usr/lib64/libgcc_s.so.1...
380: Reading symbols from .gnu_debugdata for /usr/lib64/libgcc_s.so.1...
380: (No debugging symbols found in .gnu_debugdata for /usr/lib64/libgcc_s.so.1)
380: Reading symbols from /usr/lib64/libc.so.6...
380: (No debugging symbols found in /usr/lib64/libc.so.6)
380: Reading symbols from /lib64/ld-linux-x86-64.so.2...
380: [Thread debugging using libthread_db enabled]
380: Using host libthread_db library "/usr/lib64/libthread_db.so.1".
380: 0x00007ff85833c94a in pwrite64 () from /usr/lib64/libc.so.6
380: Breakpoint 1 at 0x4f3f40: file /tmp/rr/src/RecordCommand.cc, line 635.
380:
380: Breakpoint 1, rr::force_close_record_session ()
380: at /tmp/rr/src/RecordCommand.cc:635
380: 635 if (static_session) {
380: The program being debugged stopped while in a function called from GDB.
380: Evaluation of the expression containing the function
380: (rr::force_close_record_session()) will be abandoned.
380: When the function is done executing, GDB will silently stop.
380: Detaching from program: /tmp/rr_build/bin/rr, process 367121
380: [Inferior 1 (process 367121) detached]
380: --------------------------------------------------
380: replay.out:
380: --------------------------------------------------
380: --------------------------------------------------
380: Test mmap_huge failed, leaving behind /tmp/rr-test-mmap_huge-4q5pJRGK1
380: To replay the failed test, run
380: _RR_TRACE_DIR=/tmp/rr-test-mmap_huge-4q5pJRGK1 rr replay
2/2 Test #380: mmap_huge-no-syscallbuf ..........***Failed Error regular expression found in output. Regex=[FAILED]138.39 sec
0% tests passed, 2 tests failed out of 2
Total Test time (real) = 272.27 sec
The following tests FAILED:
379 - mmap_huge (Failed)
380 - mmap_huge-no-syscallbuf (Failed)
Errors while running CTest
(same result with system gdb (GDB) Red Hat Enterprise Linux 10.2-10.el9)
Should we really run an automated test (which may also be run in parallel to the no-syscallbuf version so takes double size) that takes 1TB of virtual ram and > 58GB of memory?
I don't see this on 6.14. The test uses 1TB of virtual memory, but that's no problem at all. It hardly uses any RSS.
Can you confirm the problem is specific to a range of kernel versions?
@rocallahan it is hard to verify for me that this is kernel related, I can say that it doesn't work on this server machine with RHEL9, while it worked fine on a personal "consumer" setup under Mint with a much more recent kernel - but the whole setup is quite different, starting with the specific Intel CPU architecture. In theory I may be able to compile an old kernel on my personal computer and try if it still boots, then if it does check RRs behaviour. But man, that's a lot of work (this machine never built a kernel so far) - do you have any other idea?
Note: the machine passes everything else but pkeys, and that's kernel related #3032 and no problem as it fails within seconds, while mmap takes a lot of time and a lot of cpu to finally abort.
Just a note: I've just got to run a completely different environment (Fedora-42 on WSL2 with 6.6.87.2-microsoft-standard-WSL2 kernel on AMD Ryzen 7 3700X (cpu family 23, model 113) using rr.soft) and the only failure that is not related to rr.soft is again the mmap_huge test.
In this setup, the mmap_huge tests raise the VMs memory usage to max (16GB here), then raise the VM's io/write (swap) to 280MB/s for several seconds and then either get terminated or the whole VM gets terminated (always when running both w/wo no-syscallbuf in parallel).
The only thing that those environments have identical: they are both virtualized (but quite differently, in one case WSL2=adjusted HyperV, in the other VMWare).
Can you check the value of /proc/sys/vm/overcommit_memory on the machine(s) where it doesn't work?