libbacktrace
libbacktrace copied to clipboard
ianlancetaylor
Can't find debug info in Mach-o executable
Problem
I tried to get the stack info with libbacktrace on failure of C++ programs on mac os. My C++ code is compiled with -fno-pie, -g, but libbacktrace gave an error output: no debug info in Mach-O executable.
Environment
- mac os 10.13.6
- Apple LLVM version 10.0.0 (clang-1000.10.44.4)
Is lldb able to find the debug info? Can you find out where lldb is looking for it? Thanks.
Is lldb able to find the debug info? Can you find out where lldb is looking for it? Thanks.
yes, I have tried lldb:
vin@Mac co]$ cd build/macosx/x86_64/release/
vin@Mac release]$ lldb ./stack
(lldb) target create "./stack"
Current executable set to './stack' (x86_64).
(lldb) r
Process 11192 launched: './stack' (x86_64)
Process 11192 stopped
* thread #2, stop reason = EXC_BAD_ACCESS (code=1, address=0x0)
frame #0: 0x00000001000010ec stack`a() at stack.cc:15
12 if (FLG_check) {
13 CHECK_EQ(1 + 1, 3);
14 } else {
-> 15 *p = 'c';
16 }
17 }
18
Target 0: (stack) stopped.
(lldb) bt
* thread #2, stop reason = EXC_BAD_ACCESS (code=1, address=0x0)
* frame #0: 0x00000001000010ec stack`a() at stack.cc:15
frame #1: 0x00000001000011c9 stack`b() at stack.cc:20
frame #2: 0x00000001000011d9 stack`c() at stack.cc:24
frame #3: 0x0000000100002017 stack`co::xx::Function0p<void ()>::run(this=0x00000001003004c0) at closure.h:40
frame #4: 0x0000000100003e7f stack`co::SchedulerImpl::main_func(from=<unavailable>) at scheduler.cc:36 [opt]
frame #5: 0x0000000100002238 stack`_entry at context_x64.S:264
(lldb)
Run without lldb(I use backtrace_full() to get the stack info):
vin@Mac release]$ ./stack
F0921 11:57:30.560] SIGSEGV: segmentation fault
no debug info in Mach-O executable
errno: -1
no debug info in Mach-O executable
errno: -1
no debug info in Mach-O executable
errno: -1
no debug info in Mach-O executable
errno: -1
no debug info in Mach-O executable
errno: -1
Segmentation fault: 11
On Linux, It works well:
alvin@Envy:/mnt/d/cc/co/build/linux/x86_64/release (dev)
$ ./stack
F0921 12:02:19.256] SIGSEGV: segmentation fault
#0 in a() at test/stack.cc:15
#1 in b() at test/stack.cc:20
#2 in c() at test/stack.cc:24
#3 in co::xx::Function0p<void ()>::run() at include/co/closure.h:40
#4 in co::SchedulerImpl::main_func(__tb_context_from_t) at src/co/scheduler.cc:36
#5 in ??? at src/co/context/context_x64.S:264
Segmentation fault (core dumped)
Thanks. I don't have a macOS system. Can you run lldb under the macOS equivalent of strace to find out where it is getting the debug info? If it is not in the main executable it is probably in some related file. libbacktrace already has code to find that related file (https://github.com/ianlancetaylor/libbacktrace/blob/master/macho.c#L856). Perhaps we need an additional case there or something.
Thanks. I don't have a macOS system. Can you run lldb under the macOS equivalent of
straceto find out where it is getting the debug info? If it is not in the main executable it is probably in some related file. libbacktrace already has code to find that related file (https://github.com/ianlancetaylor/libbacktrace/blob/master/macho.c#L856). Perhaps we need an additional case there or something.
Sorry, I didn't quite get run lldb under the macOS equivalent of strace to find out where it is getting the debug info.
My test code is here.
The executable stack is built from stack.cc, and it depends on a static library libco.a.
I got the following output with lldb:
(lldb) bt
* thread #2, stop reason = EXC_BAD_ACCESS (code=1, address=0x0)
* frame #0: 0x00000001000010ec stack`a() at stack.cc:15
frame #1: 0x00000001000011c9 stack`b() at stack.cc:20
frame #2: 0x00000001000011d9 stack`c() at stack.cc:24
frame #3: 0x0000000100002017 stack`co::xx::Function0p<void ()>::run(this=0x00000001003004c0) at closure.h:40
frame #4: 0x0000000100003e7f stack`co::SchedulerImpl::main_func(from=<unavailable>) at scheduler.cc:36 [opt]
frame #5: 0x0000000100002238 stack`_entry at context_x64.S:264
frame 0-2 are from stack.cc, and frame 3-5 are from libco.a. The executable stack and libco.a are in the same directory.
Thanks, but that's not what I need to know.
The compiler puts debug info somewhere, and lldb reads the debug info in order to debug. libbacktrace also needs the same debug info. Where is the compiler putting that debug info? Where is lldb reading the debug info from?
Thanks, but that's not what I need to know.
The compiler puts debug info somewhere, and lldb reads the debug info in order to debug. libbacktrace also needs the same debug info. Where is the compiler putting that debug info? Where is lldb reading the debug info from?
Is there any command to detect that?
I am not very familiar with macOS. On Linux, I can do this:
strace -f -e openat lldb ./hello
and get output that shows me every file that lldb opens. On my system I see
openat(AT_FDCWD, "/etc/ld.so.cache", O_RDONLY|O_CLOEXEC) = 3
openat(AT_FDCWD, "/lib/x86_64-linux-gnu/libpthread.so.0", O_RDONLY|O_CLOEXEC) = 3
openat(AT_FDCWD, "/lib/x86_64-linux-gnu/liblldb-9.so.1", O_RDONLY|O_CLOEXEC) = 3
openat(AT_FDCWD, "/lib/x86_64-linux-gnu/libLLVM-9.so.1", O_RDONLY|O_CLOEXEC) = 3
openat(AT_FDCWD, "/lib/x86_64-linux-gnu/libstdc++.so.6", O_RDONLY|O_CLOEXEC) = 3
openat(AT_FDCWD, "/lib/x86_64-linux-gnu/libm.so.6", O_RDONLY|O_CLOEXEC) = 3
openat(AT_FDCWD, "/lib/x86_64-linux-gnu/libgcc_s.so.1", O_RDONLY|O_CLOEXEC) = 3
openat(AT_FDCWD, "/lib/x86_64-linux-gnu/libc.so.6", O_RDONLY|O_CLOEXEC) = 3
openat(AT_FDCWD, "/lib/x86_64-linux-gnu/../lib/tls/haswell/avx512_1/x86_64/libpython3.9.so.1.0", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/lib/x86_64-linux-gnu/../lib/tls/haswell/avx512_1/libpython3.9.so.1.0", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/lib/x86_64-linux-gnu/../lib/tls/haswell/x86_64/libpython3.9.so.1.0", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/lib/x86_64-linux-gnu/../lib/tls/haswell/libpython3.9.so.1.0", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/lib/x86_64-linux-gnu/../lib/tls/avx512_1/x86_64/libpython3.9.so.1.0", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/lib/x86_64-linux-gnu/../lib/tls/avx512_1/libpython3.9.so.1.0", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/lib/x86_64-linux-gnu/../lib/tls/x86_64/libpython3.9.so.1.0", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/lib/x86_64-linux-gnu/../lib/tls/libpython3.9.so.1.0", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/lib/x86_64-linux-gnu/../lib/haswell/avx512_1/x86_64/libpython3.9.so.1.0", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/lib/x86_64-linux-gnu/../lib/haswell/avx512_1/libpython3.9.so.1.0", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/lib/x86_64-linux-gnu/../lib/haswell/x86_64/libpython3.9.so.1.0", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/lib/x86_64-linux-gnu/../lib/haswell/libpython3.9.so.1.0", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/lib/x86_64-linux-gnu/../lib/avx512_1/x86_64/libpython3.9.so.1.0", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/lib/x86_64-linux-gnu/../lib/avx512_1/libpython3.9.so.1.0", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/lib/x86_64-linux-gnu/../lib/x86_64/libpython3.9.so.1.0", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/lib/x86_64-linux-gnu/../lib/libpython3.9.so.1.0", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/lib/x86_64-linux-gnu/libpython3.9.so.1.0", O_RDONLY|O_CLOEXEC) = 3
openat(AT_FDCWD, "/lib/x86_64-linux-gnu/libtinfo.so.6", O_RDONLY|O_CLOEXEC) = 3
openat(AT_FDCWD, "/lib/x86_64-linux-gnu/libedit.so.2", O_RDONLY|O_CLOEXEC) = 3
openat(AT_FDCWD, "/lib/x86_64-linux-gnu/libncurses.so.6", O_RDONLY|O_CLOEXEC) = 3
openat(AT_FDCWD, "/lib/x86_64-linux-gnu/libform.so.6", O_RDONLY|O_CLOEXEC) = 3
openat(AT_FDCWD, "/lib/x86_64-linux-gnu/libpanel.so.6", O_RDONLY|O_CLOEXEC) = 3
openat(AT_FDCWD, "/lib/x86_64-linux-gnu/libdl.so.2", O_RDONLY|O_CLOEXEC) = 3
openat(AT_FDCWD, "/lib/x86_64-linux-gnu/libffi.so.7", O_RDONLY|O_CLOEXEC) = 3
openat(AT_FDCWD, "/lib/x86_64-linux-gnu/libz.so.1", O_RDONLY|O_CLOEXEC) = 3
openat(AT_FDCWD, "/lib/x86_64-linux-gnu/librt.so.1", O_RDONLY|O_CLOEXEC) = 3
openat(AT_FDCWD, "/lib/x86_64-linux-gnu/libz3.so.4", O_RDONLY|O_CLOEXEC) = 3
openat(AT_FDCWD, "/lib/x86_64-linux-gnu/libexpat.so.1", O_RDONLY|O_CLOEXEC) = 3
openat(AT_FDCWD, "/lib/x86_64-linux-gnu/libutil.so.1", O_RDONLY|O_CLOEXEC) = 3
openat(AT_FDCWD, "/lib/x86_64-linux-gnu/libbsd.so.0", O_RDONLY|O_CLOEXEC) = 3
openat(AT_FDCWD, "/lib/x86_64-linux-gnu/libmd.so.0", O_RDONLY|O_CLOEXEC) = 3
openat(AT_FDCWD, "/proc/cpuinfo", O_RDONLY|O_CLOEXEC) = 3
--- SIGCHLD {si_signo=SIGCHLD, si_code=CLD_EXITED, si_pid=2898942, si_uid=17684, si_status=0, si_utime=2, si_stime=1} ---
--- SIGCHLD {si_signo=SIGCHLD, si_code=CLD_EXITED, si_pid=2898943, si_uid=17684, si_status=0, si_utime=3, si_stime=0} ---
openat(AT_FDCWD, "/lib/terminfo/x/xterm-256color", O_RDONLY) = 3
(lldb) target create "./hello"
openat(AT_FDCWD, "/usr/local/google/home/iant/hello", O_RDONLY|O_CLOEXEC) = 4
openat(AT_FDCWD, "/usr/local/google/home/iant/hello", O_RDONLY|O_CLOEXEC) = 4
openat(AT_FDCWD, "/usr/local/google/home/iant/hello", O_RDONLY|O_CLOEXEC) = 4
openat(AT_FDCWD, "/usr/local/google/home/iant/hello", O_RDONLY|O_CLOEXEC) = 4
openat(AT_FDCWD, "/usr/local/google/home/iant/hello", O_RDONLY|O_CLOEXEC) = 4
openat(AT_FDCWD, "/usr/local/google/home/iant/hello", O_RDONLY|O_CLOEXEC) = 4
openat(AT_FDCWD, "libc.so.6", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "libc.so.6", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
Current executable set to './hello' (x86_64).
openat(AT_FDCWD, "/lib/terminfo/x/xterm-256color", O_RDONLY) = 3
openat(AT_FDCWD, "/lib/terminfo/x/xterm-256color", O_RDONLY) = 5
openat(AT_FDCWD, "/usr/local/google/home/iant/.lldb/lldb-history", O_RDONLY) = 5
openat(AT_FDCWD, "/usr/local/google/home/iant/.editrc", O_RDONLY) = -1 ENOENT (No such file or directory)
Maybe I have found the problem. We used a different sysroot on mac os.
"/usr/bin/xcrun -sdk macosx clang++" -o build/macosx/x86_64/release/xx build/.objs/xx/macosx/x86_64/release/test/xx.cc.o -arch x86_64 -mmacosx-version-min=10.13 -isysroot /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.14.sdk -stdlib=libc++ -Lbuild/macosx/x86_64/release -lco -lbacktrace -lpthread -ldl -lz
The sysroot is set as the root directory of xcode sdk.
vin@Mac MacOSX10.14.sdk]$ pwd
/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.14.sdk
vin@Mac MacOSX10.14.sdk]$ ls
Entitlements.plist System/
SDKSettings.plist usr/
vin@Mac MacOSX10.14.sdk]$ ls usr
bin/ include/ lib/ libexec/ share/
vin@Mac MacOSX10.14.sdk]$ ls usr/lib/system/libunwind.tbd
usr/lib/system/libunwind.tbd
vin@Mac MacOSX10.14.sdk]$ ls usr/lib/libobjc
libobjc-trampolines.tbd libobjc.A.tbd libobjc.tbd
vin@Mac MacOSX10.14.sdk]$ ls usr/lib/libobjc*
usr/lib/libobjc-trampolines.tbd usr/lib/libobjc.tbd@
usr/lib/libobjc.A.tbd
And there is another sdk directory:
vin@Mac MacOSX10.14.sdk]$ xcrun --show-sdk-path
/Library/Developer/CommandLineTools/SDKs/MacOSX10.14.sdk
vin@Mac MacOSX10.14.sdk]$ pwd
/Library/Developer/CommandLineTools/SDKs/MacOSX10.14.sdk
vin@Mac MacOSX10.14.sdk]$ ls
Entitlements.plist SDKSettings.plist usr/
SDKSettings.json System/
vin@Mac MacOSX10.14.sdk]$ ls usr/lib/system/libunwind*
usr/lib/system/libunwind.tbd
I don't know why a different sysroot or SDK directory would matter, unless clang is storing the debug information there for some reason.
@ianlancetaylor
I got the following result with dtruss, It seems that lldb found debug info from /Library/Developer/CommandLineTools/usr/lib.
vin@Mac release]$ sudo dtruss -f lldb ./stack
PID/THRD SYSCALL(args) = return
714/0x1c70: open("/dev/dtracehelper\0", 0x2, 0xFFFFFFFFE3A4ECF0) = 3 0
714/0x1c70: ioctl(0x3, 0x80086804, 0x7FFEE3A4EC50) = 0 0
714/0x1c70: close(0x3) = 0 0
714/0x1c70: access("/AppleInternal/XBS/.isChrooted\0", 0x0, 0x0) = -1 Err#2
714/0x1c70: thread_selfid(0x0, 0x0, 0x0) = 7280 0
714/0x1c70: bsdthread_register(0x7FFF67941BEC, 0x7FFF67941BDC, 0x2000) = 1073742047 0
714/0x1c70: issetugid(0x0, 0x0, 0x0) = 0 0
714/0x1c70: mprotect(0x10C1B9000, 0x1000, 0x0) = 0 0
714/0x1c70: mprotect(0x10C1C0000, 0x1000, 0x0) = 0 0
714/0x1c70: mprotect(0x10C1C1000, 0x1000, 0x0) = 0 0
714/0x1c70: mprotect(0x10C1C8000, 0x1000, 0x0) = 0 0
714/0x1c70: mprotect(0x10C1B7000, 0x88, 0x1) = 0 0
714/0x1c70: mprotect(0x10C1C9000, 0x1000, 0x1) = 0 0
714/0x1c70: mprotect(0x10C1B7000, 0x88, 0x3) = 0 0
714/0x1c70: mprotect(0x10C1B7000, 0x88, 0x1) = 0 0
714/0x1c70: getpid(0x0, 0x0, 0x0) = 714 0
714/0x1c70: stat64("/AppleInternal/XBS/.isChrooted\0", 0x7FFEE3A4E298, 0x0) = -1 Err#2
714/0x1c70: stat64("/AppleInternal\0", 0x7FFEE3A4E330, 0x0) = -1 Err#2
714/0x1c70: csops(0x2CA, 0x7, 0x7FFEE3A4DDD0) = 0 0
714/0x1c70: sysctl([CTL_KERN, 14, 1, 714, 0, 0] (4), 0x7FFEE3A4DF18, 0x7FFEE3A4DF10, 0x0, 0x0) = 0 0
714/0x1c70: csops(0x2CA, 0x7, 0x7FFEE3A4D6C0) = 0 0
714/0x1c70: __mac_syscall(0x7FFF6794EAC6, 0x4, 0x7FFEE3A4EA38) = -1 Err#45
714/0x1c70: readlink("/var/db/xcode_select_link\0", 0x7FFEE3A4F6F0, 0x3FF) = 35 0
714/0x1c70: stat64("/Library/Developer/CommandLineTools/usr/lib/libxcrun.dylib\0", 0x7FFEE3A4EE50, 0x0) = 0 0
714/0x1c70: stat64("/\0", 0x7FFEE3A4C168, 0x0) = 0 0
714/0x1c70: getattrlist("/Library\0", 0x112D3A098, 0x7FFEE3A4DAB0) = 0 0
714/0x1c70: getattrlist("/Library/Developer\0", 0x112D3A098, 0x7FFEE3A4DAB0) = 0 0
714/0x1c70: getattrlist("/Library/Developer/CommandLineTools\0", 0x112D3A098, 0x7FFEE3A4DAB0) = 0 0
714/0x1c70: getattrlist("/Library/Developer/CommandLineTools/usr\0", 0x112D3A098, 0x7FFEE3A4DAB0) = 0 0
714/0x1c70: getattrlist("/Library/Developer/CommandLineTools/usr/lib\0", 0x112D3A098, 0x7FFEE3A4DAB0) = 0 0
714/0x1c70: getattrlist("/Library/Developer/CommandLineTools/usr/lib/libxcrun.dylib\0", 0x112D3A098, 0x7FFEE3A4DAB0) = 0 0
714/0x1c70: stat64("/Library/Developer/CommandLineTools/usr/lib/libxcrun.dylib\0", 0x7FFEE3A4E370, 0x0) = 0 0
714/0x1c70: open("/Library/Developer/CommandLineTools/usr/lib/libxcrun.dylib\0", 0x0, 0x0) = 3 0
714/0x1c70: pread(0x3, "\317\372\355\376\a\0", 0x1000, 0x0) = 4096 0
714/0x1c70: fcntl(0x3, 0x61, 0x7FFEE3A45B88) = 0 0
714/0x1c70: mmap(0x0, 0x688, 0x5, 0x1, 0x3, 0x0) = 0x10C1CA000 0
714/0x1c70: munmap(0x10C1CA000, 0x688) = 0 0
714/0x1c70: mmap(0x10C1CA000, 0x7000, 0x5, 0x12, 0x3, 0x0) = 0x10C1CA000 0
714/0x1c70: mmap(0x10C1D1000, 0x1000, 0x3, 0x12, 0x3, 0x7000) = 0x10C1D1000 0
714/0x1c70: mmap(0x10C1D2000, 0x3F80, 0x1, 0x12, 0x3, 0x8000) = 0x10C1D2000 0
714/0x1c70: close(0x3) = 0 0
714/0x1c70: __mac_syscall(0x7FFF6794EAC6, 0x4, 0x7FFEE3A4D818) = -1 Err#45
714/0x1c70: csops(0x2CA, 0xB, 0x7FFEE3A4D858) = -1 Err#34
714/0x1c70: csops(0x2CA, 0xB, 0x7FCBA9E001B0) = 0 0
714/0x1c70: geteuid(0x0, 0x0, 0x0) = 0 0
714/0x1c70: access("/var/folders/zz/zyxvpxvq6csfxvn_n0000000000000/\0", 0x0, 0x0) = 0 0
714/0x1c70: mkdir("/var/folders/zz/zyxvpxvq6csfxvn_n0000000000000/T/\0", 0x1C0, 0x0) = -1 Err#17
714/0x1c70: access("/Library/Developer/CommandLineTools\0", 0x4, 0x0) = 0 0
714/0x1c70: open("/var/folders/zz/zyxvpxvq6csfxvn_n0000000000000/T/xcrun_db\0", 0x0, 0xFFFFFFFFE3A4E14F) = 3 0
714/0x1c70: fstat64(0x3, 0x7FFEE3A4E078, 0x0) = 0 0
714/0x1c70: mmap(0x0, 0x7E, 0x1, 0x2, 0x3, 0x0) = 0x10C1D9000 0
714/0x1c70: close(0x3) = 0 0
714/0x1c70: stat64("/usr/include/sys/cdefs.h\0", 0x7FFEE3A4E568, 0x0) = 0 0
714/0x1c70: stat64("/Library/Developer/CommandLineTools/usr/bin/lldb\0", 0x7FFEE3A4E5C0, 0x0) = 0 0
714/0x1c70: getrlimit(0x1008, 0x7FFEE3A4E3F0, 0x0) = 0 0
714/0x1c70: open_nocancel("/Library/Developer/CommandLineTools/usr/bin/lldb\0", 0x0, 0x1B6) = 3 0
714/0x1c70: fstat64(0x3, 0x7FFEE3A4E388, 0x0) = 0 0
714/0x1c70: read_nocancel(0x3, "\317\372\355\376\a\0", 0x1000) = 4096 0
714/0x1c70: lseek(0x3, 0x0, 0x1) = 4096 0
714/0x1c70: close_nocancel(0x3) = 0 0
714/0x1c70: munmap(0x10C1D9000, 0x7E) = 0 0
@ianlancetaylor
More details can be found here: https://github.com/idealvin/tmp/blob/master/stack.log.
The above stack.log is the result of sudo dtruss -f ./x.sh.
x.sh is as follow:
#!/bin/bash
lldb -b -o r -k bt -- ./stack -co_sched_num=1