No line numbers with Clang due to missing DWARF .debug_aranges section

[Optiligence]

when i compile a program with msys2 based gcc or clang built with -g and run it in gdb, symbols are resolved correctly to file with line numbers

but when i observe a crash with Dr. Mingw i only get line numbers and the source file content with gcc, but not with clang

i think this used to work and i’m not sure what clang would be doing differently now

what can i check to discover differences and why does it work in gdb but not Dr. Mingw?

[jrfonseca]

Perhaps recent msys2 clang is generating newer DWARF information.

I'll try to update the bundled dwarf library to a newer version. Perhaps that solves the issue.

[jrfonseca]

@Optiligence , please retry with the binaries from https://github.com/jrfonseca/drmingw/releases/tag/0.9.2 that should (hopefully) become available in 15 mins or so.

[Optiligence]

hm, it doesn’t seem to work, still

i also didn’t think that they would update the DWARF spec that often (and in a breaking manner)

do you have a command at hand that displays relevant sections of the binary so i can compare them between gcc and clang or post them here?

[jrfonseca]

It had been 3 years since I last updated libdwarf. And it did made a difference in the past, be from gcc using newer dwarf features or bugs.

Please build the sample from https://github.com/jrfonseca/drmingw/tree/master/sample with MSYS2 Clang, and confirm it doesn't work. If that sample is too trivial to repro the issue, then please modify it to be something more complex that repros.

Also confirm the clang version (clang++ --version).

I'll then repro locally myself and try to figure out what's going on.

Another interesting experiment is confirming whether the problem is the C++ compiler or the linker (as I understand clang/llvm are moving to lld.) So it would be good to know if compiling with G++ and linking with clang++ repros the issue or not, and vice versa, compiling with clang++ and linking with g++.

[Optiligence]

i commented force_debug ()

g++.exe (Rev3, Built by MSYS2 project) 9.1.0 Copyright (C) 2019 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

cmake -G Ninja .. -DEXCHNDL_IMPLIB='/c/Program Files/drmingw-0.9.2-win64/lib/libexchndl.a' -DCMAKE_BUILD_TYPE=DEBUG

[1/2] D:\msys64\mingw64\bin\c++.exe -g -MD -MT CMakeFiles/sample.dir/sample.obj -MF CMakeFiles\sample.dir\sample.obj.d -o CMakeFiles/sample.dir/sample.obj -c ../sample.cpp [2/2] cmd.exe /C "cd . && D:\msys64\mingw64\bin\c++.exe -g CMakeFiles/sample.dir/sample.obj -o sample.exe -Wl,--major-image-version,0,--minor-image-version,0 "C:/Program Files/drmingw-0.9.2-win64/lib/libexchndl.a" -lkernel32 -luser32 -lgdi32 -lwinspool -lshell32 -lole32 -loleaut32 -luuid -lcomdlg32 -ladvapi32 && cd ."

sample.exe caused an Access Violation at location 00007FFBCD3F4715 in module msvcrt.dll Writing to location 0000000000000001.

AddrPC           Params
00007FFBCD3F4715 0000000000000000 000000000000002C 0000000000000000  msvcrt.dll!_input_l
00007FFBCD3E9C14 0000000000FE6F60 0000000000000000 0000000000407028  msvcrt.dll!vscan_fn
00007FFBCD3E9B80 0000000000407028 0000000000404000 0000000000000001  msvcrt.dll!sscanf
000000000040153D 0000000000000008 4016666660000000 0000000000404009  sample.exe!Function  [C:/dev/dr.mingw sample/build_gcc/../sample.cpp @ 12]
    10: 
    11: static void Function(int i, double j, const char * pszString) {
>   12:     sscanf("12345", "%i", (int *)1);
    13: }
    14: 
0000000000402B9C 0000000000000004 0000000040B33333 00000000004015A0  sample.exe!StaticMethod  [C:/dev/dr.mingw sample/build_gcc/../sample.cpp @ 17]
    15: struct Class {
    16:     static void StaticMethod(int i, float j) {
>   17:         Function(i * 2, j, "Hello");
    18:     }
    19: 
0000000000402BCE 000000000065FE1F 00000000004014D9 0000000000000001  sample.exe!Method  [C:/dev/dr.mingw sample/build_gcc/../sample.cpp @ 21]
    19: 
    20:     void Method(void) {
>   21:         StaticMethod(4, 5.6f);
    22:     }
    23: };
0000000000401562 0000000000000000 000000000000002C 0000000000407970  sample.exe!main  [C:/dev/dr.mingw sample/build_gcc/../sample.cpp @ 28]
    26:     ExcHndlInit();
    27:     Class instance;
>   28:     instance.Method();
    29:     return 0;
    30: }
000000000040138C 0000000000000000 0000000000000000 0000000000000000  sample.exe!__tmainCRTStartup  [E:/mingwbuild/mingw-w64-crt-git/src/mingw-w64/mingw-w64-crt/crt/crtexe.c @ 339]
00000000004014BB 0000000000000000 0000000000000000 0000000000000000  sample.exe!mainCRTStartup  [E:/mingwbuild/mingw-w64-crt-git/src/mingw-w64/mingw-w64-crt/crt/crtexe.c @ 223]
00007FFBCDF97BD4 0000000000000000 0000000000000000 0000000000000000  KERNEL32.DLL!BaseThreadInitThunk
00007FFBCE46CE71 0000000000000000 0000000000000000 0000000000000000  ntdll.dll!RtlUserThreadStart

---

clang version 8.0.1 (tags/RELEASE_801/final) Target: x86_64-w64-windows-gnu Thread model: posix InstalledDir: D:\msys64\mingw64\bin

cmake -G Ninja .. -DEXCHNDL_IMPLIB='/c/Program Files/drmingw-0.9.2-win64/lib/libexchndl.a' -DCMAKE_BUILD_TYPE=DEBUG -DCMAKE_CXX_COMPILER=clang++

[1/2] D:\msys64\mingw64\bin\clang++.exe -g -MD -MT CMakeFiles/sample.dir/sample.obj -MF CMakeFiles\sample.dir\sample.obj.d -o CMakeFiles/sample.dir/sample.obj -c ../sample.cpp [2/2] cmd.exe /C "cd . && D:\msys64\mingw64\bin\clang++.exe -g CMakeFiles/sample.dir/sample.obj -o sample.exe -Wl,--major-image-version,0,--minor-image-version,0 "C:/Program Files/drmingw-0.9.2-win64/lib/libexchndl.a" -lkernel32 -luser32 -lgdi32 -lwinspool -lshell32 -lole32 -loleaut32 -luuid -lcomdlg32 -ladvapi32 && cd ."

sample.exe caused an Access Violation at location 00007FFBCD3F4715 in module msvcrt.dll Writing to location 0000000000000001.

AddrPC           Params
00007FFBCD3F4715 0000000000000000 00007FFBCDF9A0D0 0000000000000000  msvcrt.dll!_input_l
00007FFBCD3E9C14 000000000000002E 0000000000000040 00000000001E0350  msvcrt.dll!vscan_fn
00007FFBCD3E9B80 0000000000000000 0000000000404010 0000000000000001  msvcrt.dll!sscanf
000000000040156C 0000000000401500 0000000000000008 00000000001E6F90  sample.exe!Function
0000000000402B9B 0000000000000001 0000000000000000 00000000004015A0  sample.exe!Class::StaticMethod
0000000000402BCB 00000000001E6FB0 00000000004014D9 0000000000000001  sample.exe!Class::Method
0000000000401534 0000000000000000 000000000000002E 0000000000407970  sample.exe!main
000000000040138C 0000000000000000 0000000000000000 0000000000000000  sample.exe!__tmainCRTStartup  [E:/mingwbuild/mingw-w64-crt-git/src/mingw-w64/mingw-w64-crt/crt/crtexe.c @ 339]
00000000004014BB 0000000000000000 0000000000000000 0000000000000000  sample.exe!mainCRTStartup  [E:/mingwbuild/mingw-w64-crt-git/src/mingw-w64/mingw-w64-crt/crt/crtexe.c @ 223]
00007FFBCDF97BD4 0000000000000000 0000000000000000 0000000000000000  KERNEL32.DLL!BaseThreadInitThunk
00007FFBCE46CE71 0000000000000000 0000000000000000 0000000000000000  ntdll.dll!RtlUserThreadStart

[Optiligence]

i think they both use GNU ld by default

[jrfonseca]

The problem is that dwarf_get_arange call in src/mgwhelp/dwarf_find.cpp is failing for clang.

I'm not sure why. I suspect that the addresses emitted by clang have a different base offset.

When I use MinGW addr2line tool, it handles clang binaries fine.

Perhaps there's some relate offset for debug addresses (or flag) somewhere in DWARF info or PE I'm neglecting to .

The only reason there's any symbols, is because mgwhelp is falling back to the PE's symbol table (it's not using DWARF at all.)

If you have the ability/opportunity to investigate further it would be appreciated.

[Optiligence]

@mati865 sorry to drag you in like this

you don’t happen to have a quick helpful insight here?

[mati865]

Sorry but I hardly know anything about debug symbols in Clang.

You could try with this toolchain (there are binaries in Releases tab): https://github.com/mstorsjo/llvm-mingw Mstorsjo is MinGW maintainer for LLVM project so ping him if it doesn't work with his toolchain.

[Optiligence]

Thanks @mati865

It also doesn’t work with his toolchain. (and that one uses lld)

sample.exe caused an Access Violation at location 00007FF94BDF8547 in module ucrtbase.dll Writing to location 0000000000000001.

AddrPC           Params
00007FF94BDF8547 000000000014FC01 000000000014FCA9 000000000014FC40  ucrtbase.dll!__crt_stdio_input::input_processor<char,__crt_stdio_input::string_input_adapter<char> >::write_integer
00007FF94BDF7113 0000000001000001 000000000014FC40 0000000140004010  ucrtbase.dll!__crt_stdio_input::input_processor<char,__crt_stdio_input::string_input_adapter<char> >::process_integer_specifier
00007FF94BDF736B 000000000014FC40 0075006C006F0056 005C00310065006D  ucrtbase.dll!__crt_stdio_input::input_processor<char,__crt_stdio_input::string_input_adapter<char> >::process_state
00007FF94BDF6BD2 0000000000000005 0000000000000005 006C006900750062  ucrtbase.dll!__crt_stdio_input::input_processor<char,__crt_stdio_input::string_input_adapter<char> >::process
00007FF94BDF12B2 0000000000000001 00007FF94BD89FDE 0000000000000000  ucrtbase.dll!common_vsscanf<char>
0000000140002C82 000000000014FDF0 00007FF94BE6BC20 0000000000000001  sample.exe!sscanf
000000014000150C 000000000014FDE0 000000000014FDE8 000000000000002F  sample.exe!Function
0000000140002BED 0000000000000001 000000014000236D 00007FF94BE6C028  sample.exe!Class::StaticMethod
0000000140002C1B 0000000000437B20 0000000140001489 0000000000000002  sample.exe!Class::Method
00000001400014D4 0000000000000000 000000014000711C 0000000000000000  sample.exe!main
000000014000140A 0000000000000000 0000000000000000 0000000000000000  sample.exe!__tmainCRTStartup
000000014000146B 0000000000000000 0000000000000000 0000000000000000  sample.exe!mainCRTStartup
00007FF94E9D7BD4 0000000000000000 0000000000000000 0000000000000000  KERNEL32.DLL!BaseThreadInitThunk
00007FF94EBACE71 0000000000000000 0000000000000000 0000000000000000  ntdll.dll!RtlUserThreadStart

@mstorsjo do you have a comment on this?

[jrfonseca]

It seems the problem is that clang does not generate DWARF .debug_aranges information.

There was a patch to add and option for this, http://lists.llvm.org/pipermail/cfe-commits/Week-of-Mon-20140210/099034.html , but the MSYS2 version doesn't seem to support it.

Per http://wiki.dwarfstd.org/index.php?title=Best_Practices#Generating_.debug_aranges_data it sounds like generating .debug_aranges is not strictly necessary, and that drmingw should probably walk the compilation units instead..

Per https://github.com/rust-lang/rust/issues/45246 this seems an issue with other Clang based languages and other tools.

A solution is to pass -mllvm -generate-arange-section to clang. With that, clang generated binaries work.

If somebody wants to submit a patch that makes drmingw to resolve symbols, even without debug_aranges data, I'd be happy to accept it. But I don't plan to do this myself. I'll leave this issue open for documentation.

[jrfonseca]

I think a solution is to iterate over compilation units, like done in https://github.com/elftoolchain/elftoolchain/blob/trunk/addr2line/addr2line.c

[Optiligence]

Both -mllvm -generate-arange-section as well as -gdwarf-aranges seem to work.

Although it is not stated as required, it is advocated for in the DWARF wiki. For compatibility reasons it would be good if drmingw would work without it, but it seems to me that generating it should be the default or even be required by the DWARF spec. (idk much about it, but when generating debug information it’s not like you’re that concerned about size anyway)

[wtfah19]

great