server-edition
server-edition copied to clipboard
Published
20 hours ago •
fullstaq-ruby
fullstaq-ruby
Missing debugging symbols
Describe the bug
We're trying to diagnose some hung processes. We're able to use signal handling to get ruby backtraces (ie sidekiq's TTIN handler), but the problem seems to be one level deeper, ie C. The processes are multithreaded, so we need to get the state of every
SIGABRT/SIGEV are able to trigger ruby dumping it's own
These signals (and others) do also trigger a kernel core dump. If I try to analyze the core with gdb after the fact, it doesn't appear there are any debugging symbols present:
Reading symbols from /usr/lib/fullstaq-ruby/versions/3.2.2-jemalloc/bin/ruby...
(No debugging symbols found in /usr/lib/fullstaq-ruby/versions/3.2.2-jemalloc/bin/ruby)
To Reproduce
Linux distribution: Ubuntu 22.04
Package name: fullstaq-ruby-3.2.2
Steps to reproduce the behavior:
- Enable dumping cores:
ulimit -c unlimited - Run command
ruby -e 'puts Process.pid; sleep 10000' &, thenkill -s ABRT <pid> -
gdb $(which ruby) core(assumingkernel.core_pattern=core)
Note: this is running an x86_64 docker image on an M1, so I a fair amount of output is due to that rather than debugging symbols.
Reading symbols from /usr/lib/fullstaq-ruby/versions/3.2.2-jemalloc/bin/ruby...
(No debugging symbols found in /usr/lib/fullstaq-ruby/versions/3.2.2-jemalloc/bin/ruby)
warning: malformed note - filename area is too big
warning: Can't open file @ which was expanded to @ during file-backed mapping note processing
warning: Can't open file which was expanded to during file-backed mapping note processing
warning: Can't open file which was expanded to during file-backed mapping note processing
warning: Can't open file which was expanded to during file-backed mapping note processing
warning: Can't open file `@ which was expanded to `@ during file-backed mapping note processing
warning: Can't open file which was expanded to during file-backed mapping note processing
warning: Can't open file which was expanded to during file-backed mapping note processing
warning: Can't open file ( which was expanded to ( during file-backed mapping note processing
warning: Can't open file which was expanded to during file-backed mapping note processing
warning: Can't open file which was expanded to during file-backed mapping note processing
warning: Can't open file which was expanded to during file-backed mapping note processing
warning: Can't open file which was expanded to during file-backed mapping note processing
warning: Can't open file which was expanded to during file-backed mapping note processing
warning: Can't open file which was expanded to during file-backed mapping note processing
warning: Can't open file which was expanded to during file-backed mapping note processing
warning: Can't open file `@ which was expanded to `@ during file-backed mapping note processing
warning: Can't open file which was expanded to during file-backed mapping note processing
warning: Can't open file which was expanded to during file-backed mapping note processing
warning: Can't open file which was expanded to during file-backed mapping note processing
warning: Can't open file �!@ which was expanded to �!@ during file-backed mapping note processing
warning: Can't open file which was expanded to during file-backed mapping note processing
warning: Can't open file which was expanded to during file-backed mapping note processing
warning: Can't open file � which was expanded to � during file-backed mapping note processing
warning: Can't open file which was expanded to during file-backed mapping note processing
warning: Can't open file which was expanded to during file-backed mapping note processing
warning: Can't open file which was expanded to during file-backed mapping note processing
warning: Can't open file which was expanded to during file-backed mapping note processing
warning: Can't open file which was expanded to during file-backed mapping note processing
warning: Can't open file which was expanded to during file-backed mapping note processing
warning: Can't open file �!@ which was expanded to �!@ during file-backed mapping note processing
warning: Can't open file which was expanded to during file-backed mapping note processing
warning: Can't open file which was expanded to during file-backed mapping note processing
warning: Can't open file which was expanded to during file-backed mapping note processing
warning: Can't open file "@ which was expanded to "@ during file-backed mapping note processing
warning: Can't open file which was expanded to during file-backed mapping note processing
warning: Can't open file which was expanded to during file-backed mapping note processing
warning: Can't open file which was expanded to during file-backed mapping note processing
warning: Can't open file which was expanded to during file-backed mapping note processing
warning: Can't open file which was expanded to during file-backed mapping note processing
warning: Can't open file which was expanded to during file-backed mapping note processing
warning: Can't open file which was expanded to during file-backed mapping note processing
warning: Can't open file which was expanded to during file-backed mapping note processing
warning: Can't open file which was expanded to during file-backed mapping note processing
warning: Can't open file "@ which was expanded to "@ during file-backed mapping note processing
warning: Can't open file which was expanded to during file-backed mapping note processing
warning: Can't open file which was expanded to during file-backed mapping note processing
warning: Can't open file which was expanded to during file-backed mapping note processing
warning: Can't open file @"@ which was expanded to @"@ during file-backed mapping note processing
warning: Can't open file which was expanded to during file-backed mapping note processing
warning: Can't open file which was expanded to during file-backed mapping note processing
warning: Can't open file which was expanded to during file-backed mapping note processing
warning: Can't open file which was expanded to during file-backed mapping note processing
warning: Can't open file which was expanded to during file-backed mapping note processing
warning: Can't open file which was expanded to during file-backed mapping note processing
warning: Can't open file which was expanded to during file-backed mapping note processing
warning: Can't open file which was expanded to during file-backed mapping note processing
warning: Can't open file which was expanded to during file-backed mapping note processing
warning: Can't open file 0#@ which was expanded to 0#@ during file-backed mapping note processing
warning: Can't open file which was expanded to during file-backed mapping note processing
warning: Can't open file which was expanded to during file-backed mapping note processing
warning: Can't open file which was expanded to during file-backed mapping note processing
warning: Can't open file `#@ which was expanded to `#@ during file-backed mapping note processing
warning: Can't open file which was expanded to during file-backed mapping note processing
warning: Can't open file which was expanded to during file-backed mapping note processing
warning: Can't open file which was expanded to during file-backed mapping note processing
warning: Can't open file which was expanded to during file-backed mapping note processing
warning: Can't open file which was expanded to during file-backed mapping note processing
warning: Can't open file which was expanded to during file-backed mapping note processing
warning: Can't open file which was expanded to during file-backed mapping note processing
warning: Can't open file which was expanded to during file-backed mapping note processing
warning: Can't open file which was expanded to during file-backed mapping note processing
warning: Can't open file which was expanded to during file-backed mapping note processing
warning: Can't open file which was expanded to during file-backed mapping note processing
warning: Can't open file `#@ which was expanded to `#@ during file-backed mapping note processing
warning: Can't open file which was expanded to during file-backed mapping note processing
warning: Can't open file which was expanded to during file-backed mapping note processing
warning: Can't open file which was expanded to during file-backed mapping note processing
warning: Can't open file �$@ which was expanded to �$@ during file-backed mapping note processing
warning: Can't open file which was expanded to during file-backed mapping note processing
warning: Can't open file which was expanded to during file-backed mapping note processing
warning: Can't open file which was expanded to during file-backed mapping note processing
warning: Can't open file which was expanded to during file-backed mapping note processing
warning: Can't open file which was expanded to during file-backed mapping note processing
warning: Can't open file which was expanded to during file-backed mapping note processing
warning: Can't open file which was expanded to during file-backed mapping note processing
warning: Can't open file which was expanded to during file-backed mapping note processing
warning: Can't open file which was expanded to during file-backed mapping note processing
warning: Can't open file which was expanded to during file-backed mapping note processing
warning: Can't open file �$@ which was expanded to �$@ during file-backed mapping note processing
warning: core file may not match specified executable file.
warning: Selected architecture i386:x86-64 is not compatible with reported target architecture i386
warning: Architecture rejected target-supplied description
warning: Couldn't find general-purpose registers in core file.
warning: Unexpected size of section `.reg2' in core file.
Core was generated by `/usr/bin/qemu-x86_64 /usr/lib/fullstaq-ruby/versions/3.2.2-jemalloc/bin/ruby ru'.
warning: Couldn't find general-purpose registers in core file.
warning: Unexpected size of section `.reg2' in core file.
#0 <unavailable> in ?? ()
(gdb)
Expected behavior
After opening the core, being able to run bt and see the backtrace. This is what it looks like on the vanilla ruby:3.2.2 docker image:
GNU gdb (Debian 10.1-1.7) 10.1.90.20210103-git
Copyright (C) 2021 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Type "show copying" and "show warranty" for details.
This GDB was configured as "aarch64-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<https://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
<http://www.gnu.org/software/gdb/documentation/>.
For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from /usr/local/bin/ruby...
[New LWP 370]
[New LWP 371]
[New LWP 373]
[New LWP 372]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/aarch64-linux-gnu/libthread_db.so.1".
Core was generated by `ruby reproducible-case.rb'.
Program terminated with signal SIGABRT, Aborted.
#0 0x0000fffface5b3c8 in __ppoll (fds=fds@entry=0xfffff4c4d190, nfds=nfds@entry=1, timeout=<optimized out>, timeout@entry=0x0, sigmask=sigmask@entry=0x0)
at ../sysdeps/unix/sysv/linux/ppoll.c:44
44 ../sysdeps/unix/sysv/linux/ppoll.c: No such file or directory.
[Current thread is 1 (Thread 0xffffacd6da00 (LWP 370))]
(gdb) bt
#0 0x0000fffface5b3c8 in __ppoll (fds=fds@entry=0xfffff4c4d190, nfds=nfds@entry=1, timeout=<optimized out>, timeout@entry=0x0, sigmask=sigmask@entry=0x0)
at ../sysdeps/unix/sysv/linux/ppoll.c:44
#1 0x0000ffffad3ee258 in rb_sigwait_sleep (th=th@entry=0xaaaae7b964f0, sigwait_fd=sigwait_fd@entry=3, rel=rel@entry=0x0) at hrtime.h:158
#2 0x0000ffffad3ef6c8 [PAC] in native_sleep (th=th@entry=0xaaaae7b964f0, rel=0x0) at thread_pthread.c:2347
#3 0x0000ffffad3f3090 [PAC] in thread_join_sleep (arg=arg@entry=281474788283432) at thread.c:1056
#4 0x0000ffffad263d10 [PAC] in rb_ensure (b_proc=b_proc@entry=0xffffad3f2e90 <thread_join_sleep>, data1=data1@entry=281474788283432,
e_proc=e_proc@entry=0xffffad3e7ee0 <remove_from_join_list>, data2=data2@entry=281474788283432) at eval.c:1007
#5 0x0000ffffad3eb0c8 [PAC] in thread_join (limit=0x0, timeout=4, target_th=0xaaaae7fa66f0) at thread.c:1105
#6 thread_join_m (argc=<optimized out>, argv=<optimized out>, self=<optimized out>) at thread.c:1202
#7 0x0000ffffad444770 [PAC] in vm_call0_cfunc_with_frame (argv=0xfffff4c4d650, calling=0xfffff4c4d4c8, ec=0xaaaae7b97680) at vm_eval.c:150
#8 vm_call0_cfunc (argv=0xfffff4c4d650, calling=0xfffff4c4d4c8, ec=0xaaaae7b97680) at vm_eval.c:164
#9 vm_call0_body (ec=ec@entry=0xaaaae7b97680, calling=calling@entry=0xfffff4c4d598, argv=argv@entry=0xfffff4c4d650) at vm_eval.c:210
#10 0x0000ffffad4462c8 [PAC] in vm_call0_cc (kw_splat=0, cc=0xffffa8466ad8, argv=0xfffff4c4d650, argc=0, id=13617, recv=281473504931160, ec=0xaaaae7b97680) at vm_eval.c:87
#11 rb_call0 (ec=0xaaaae7b97680, recv=281473504931160, mid=13617, argc=argc@entry=0, argv=argv@entry=0xfffff4c4d650, call_scope=call_scope@entry=CALL_PUBLIC, self=281473504931360)
at vm_eval.c:551
#12 0x0000ffffad446fcc [PAC] in rb_call (scope=CALL_PUBLIC, argv=0xfffff4c4d650, argc=0, mid=<optimized out>, recv=<optimized out>) at vm_eval.c:877
#13 0x0000ffffad3d9524 in rb_sym_proc_call (mid=<optimized out>, argc=argc@entry=1, argv=argv@entry=0xfffff4c4d648, kw_splat=kw_splat@entry=0, passed_proc=passed_proc@entry=4)
at string.c:11547
#14 0x0000ffffad4439dc in vm_yield_with_symbol (block_handler=0, kw_splat=0, argv=0xfffff4c4d648, argc=1, symbol=<optimized out>, ec=0xaaaae7b97680) at vm_insnhelper.c:4297
#15 invoke_block_from_c_bh (force_blockarg=<optimized out>, is_lambda=<optimized out>, cref=<optimized out>, passed_block_handler=<optimized out>, kw_splat=<optimized out>,
argv=<optimized out>, argc=<optimized out>, block_handler=<optimized out>, ec=<optimized out>) at vm.c:1481
#16 vm_yield_with_cref (is_lambda=0, cref=0x0, kw_splat=0, argv=0xfffff4c4d648, argc=1, ec=0xaaaae7b97680) at vm.c:1509
#17 vm_yield (kw_splat=0, argv=0xfffff4c4d648, argc=1, ec=0xaaaae7b97680) at vm.c:1517
#18 rb_yield_0 (argv=0xfffff4c4d648, argc=1) at vm_eval.c:1348
#19 rb_yield (val=<optimized out>) at vm_eval.c:1364
#20 0x0000ffffad1c6bd4 [PAC] in rb_ary_each (ary=281473504931360) at array.c:2740
#21 0x0000ffffad42a120 [PAC] in vm_call_cfunc_with_frame (ec=0xaaaae7b97680, reg_cfp=0xffffacd16f90, calling=<optimized out>) at vm_insnhelper.c:3268
#22 0x0000ffffad4300f8 [PAC] in vm_sendish (ec=0xaaaae7b97680, reg_cfp=0xffffacd16f90, cd=0xaaaae7c522b0, block_handler=<optimized out>, method_explorer=mexp_search_method)
at vm_callinfo.h:367
#23 0x0000ffffad439970 [PAC] in vm_exec_core (ec=0xaaaae7b97680, initial=1, initial@entry=0) at insns.def:801
#24 0x0000ffffad43f4fc [PAC] in rb_vm_exec (ec=0xaaaae7b97680, jit_enable_p=true) at vm.c:2383
#25 0x0000ffffad25f54c [PAC] in rb_ec_exec_node (ec=ec@entry=0xaaaae7b97680, n=n@entry=0xffffacc00b00) at eval.c:289
#26 0x0000ffffad2659bc [PAC] in ruby_run_node (n=0xffffacc00b00) at eval.c:330
#27 0x0000aaaab8f90b9c [PAC] in rb_main (argv=0xfffff4c4dd68, argc=2) at ./main.c:38
#28 main (argc=<optimized out>, argv=<optimized out>) at ./main.c:57
Additional context
Interestingly, I am able to get C level backtrace when using kill -s SEGV. However, it seems to only include the backtrace of the thread responding, not multiple threads.
