llvm-project
llvm-project copied to clipboard
llvm
LLDB: Breakpoint/jit-loader* tests fail for me (on Linux and FreeBSD)
This is a recent regression, I'm going to try bisecting it later:
FAIL: lldb-shell :: Breakpoint/jit-loader_rtdyld_elf.test (16 of 17)
******************** TEST 'lldb-shell :: Breakpoint/jit-loader_rtdyld_elf.test' FAILED ********************
Script:
--
: 'RUN: at line 6'; /home/mgorny/git/llvm-project/build/bin/clang --target=specify-a-target-or-use-a-_host-substitution -g -S -emit-llvm --target=x86_64-unknown-unknown-elf -o /home/mgorny/git/llvm-project/build/tools/lldb/test/Shell/Breakpoint/Output/jit-loader_rtdyld_elf.test.tmp.ll /home/mgorny/git/llvm-project/lldb/test/Shell/Breakpoint/Inputs/jitbp.cpp
: 'RUN: at line 9'; /home/mgorny/git/llvm-project/build/bin/lldb --no-lldbinit -S /home/mgorny/git/llvm-project/build/tools/lldb/test/Shell/lit-lldb-init -b -o 'settings set plugin.jit-loader.gdb.enable on' -o 'b jitbp' -o 'run --jit-kind=mcjit /home/mgorny/git/llvm-project/build/tools/lldb/test/Shell/Breakpoint/Output/jit-loader_rtdyld_elf.test.tmp.ll' /home/mgorny/git/llvm-project/build/bin/lli | /home/mgorny/git/llvm-project/build/bin/FileCheck /home/mgorny/git/llvm-project/lldb/test/Shell/Breakpoint/jit-loader_rtdyld_elf.test
: 'RUN: at line 12'; /home/mgorny/git/llvm-project/build/bin/lldb --no-lldbinit -S /home/mgorny/git/llvm-project/build/tools/lldb/test/Shell/lit-lldb-init -b -o 'settings set plugin.jit-loader.gdb.enable on' -o 'b jitbp' -o 'run --jit-linker=rtdyld /home/mgorny/git/llvm-project/build/tools/lldb/test/Shell/Breakpoint/Output/jit-loader_rtdyld_elf.test.tmp.ll' /home/mgorny/git/llvm-project/build/bin/lli | /home/mgorny/git/llvm-project/build/bin/FileCheck /home/mgorny/git/llvm-project/lldb/test/Shell/Breakpoint/jit-loader_rtdyld_elf.test
--
Exit Code: 1
Command Output (stderr):
--
/home/mgorny/git/llvm-project/lldb/test/Shell/Breakpoint/jit-loader_rtdyld_elf.test:17:10: error: CHECK: expected string not found in input
# CHECK: Process {{.*}} stopped
^
<stdin>:17:138: note: scanning from here
(lldb) run --jit-kind=mcjit /home/mgorny/git/llvm-project/build/tools/lldb/test/Shell/Breakpoint/Output/jit-loader_rtdyld_elf.test.tmp.ll
^
<stdin>:18:1: note: possible intended match here
Process 536389 exited with status = 0 (0x00000000)
^
Input file: <stdin>
Check file: /home/mgorny/git/llvm-project/lldb/test/Shell/Breakpoint/jit-loader_rtdyld_elf.test
-dump-input=help explains the following input dump.
Input was:
<<<<<<
1: (lldb) command source -s 0 '/home/mgorny/git/llvm-project/build/tools/lldb/test/Shell/lit-lldb-init'
2: Executing commands in '/home/mgorny/git/llvm-project/build/tools/lldb/test/Shell/lit-lldb-init'.
3: (lldb) # LLDB init file for the LIT tests.
4: (lldb) settings set symbols.enable-external-lookup false
5: (lldb) settings set plugin.process.gdb-remote.packet-timeout 60
6: (lldb) settings set interpreter.echo-comment-commands false
7: (lldb) settings set symbols.clang-modules-cache-path "/home/mgorny/git/llvm-project/build/lldb-test-build.noindex/module-cache-lldb"
8: (lldb) settings set target.auto-apply-fixits false
9: (lldb) settings set target.inherit-tcc true
10: (lldb) settings set target.detach-on-error false
11: (lldb) target create "/home/mgorny/git/llvm-project/build/bin/lli"
12: Current executable set to '/home/mgorny/git/llvm-project/build/bin/lli' (x86_64).
13: (lldb) settings set plugin.jit-loader.gdb.enable on
14: (lldb) b jitbp
15: Breakpoint 1: no locations (pending).
16: WARNING: Unable to resolve breakpoint to any actual locations.
17: (lldb) run --jit-kind=mcjit /home/mgorny/git/llvm-project/build/tools/lldb/test/Shell/Breakpoint/Output/jit-loader_rtdyld_elf.test.tmp.ll
check:17'0 X error: no match found
18: Process 536389 exited with status = 0 (0x00000000)
check:17'0 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
check:17'1 ? possible intended match
19: Process 536389 launched: '/home/mgorny/git/llvm-project/build/bin/lli' (x86_64)
check:17'0 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
>>>>>>
--
********************
FAIL: lldb-shell :: Breakpoint/jit-loader_jitlink_elf.test (17 of 17)
******************** TEST 'lldb-shell :: Breakpoint/jit-loader_jitlink_elf.test' FAILED ********************
Script:
--
: 'RUN: at line 6'; /home/mgorny/git/llvm-project/build/bin/clang --target=specify-a-target-or-use-a-_host-substitution -g -S -emit-llvm -fPIC --target=x86_64-unknown-unknown-elf -o /home/mgorny/git/llvm-project/build/tools/lldb/test/Shell/Breakpoint/Output/jit-loader_jitlink_elf.test.tmp.ll /home/mgorny/git/llvm-project/lldb/test/Shell/Breakpoint/Inputs/jitbp.cpp
: 'RUN: at line 8'; /home/mgorny/git/llvm-project/build/bin/lldb --no-lldbinit -S /home/mgorny/git/llvm-project/build/tools/lldb/test/Shell/lit-lldb-init -b -o 'settings set plugin.jit-loader.gdb.enable on' -o 'b jitbp' -o 'run --jit-linker=jitlink /home/mgorny/git/llvm-project/build/tools/lldb/test/Shell/Breakpoint/Output/jit-loader_jitlink_elf.test.tmp.ll' /home/mgorny/git/llvm-project/build/bin/lli | /home/mgorny/git/llvm-project/build/bin/FileCheck /home/mgorny/git/llvm-project/lldb/test/Shell/Breakpoint/jit-loader_jitlink_elf.test
--
Exit Code: 1
Command Output (stderr):
--
/home/mgorny/git/llvm-project/lldb/test/Shell/Breakpoint/jit-loader_jitlink_elf.test:13:10: error: CHECK: expected string not found in input
# CHECK: Process {{.*}} stopped
^
<stdin>:17:143: note: scanning from here
(lldb) run --jit-linker=jitlink /home/mgorny/git/llvm-project/build/tools/lldb/test/Shell/Breakpoint/Output/jit-loader_jitlink_elf.test.tmp.ll
^
<stdin>:18:1: note: possible intended match here
Process 536387 exited with status = 0 (0x00000000)
^
Input file: <stdin>
Check file: /home/mgorny/git/llvm-project/lldb/test/Shell/Breakpoint/jit-loader_jitlink_elf.test
-dump-input=help explains the following input dump.
Input was:
<<<<<<
1: (lldb) command source -s 0 '/home/mgorny/git/llvm-project/build/tools/lldb/test/Shell/lit-lldb-init'
2: Executing commands in '/home/mgorny/git/llvm-project/build/tools/lldb/test/Shell/lit-lldb-init'.
3: (lldb) # LLDB init file for the LIT tests.
4: (lldb) settings set symbols.enable-external-lookup false
5: (lldb) settings set plugin.process.gdb-remote.packet-timeout 60
6: (lldb) settings set interpreter.echo-comment-commands false
7: (lldb) settings set symbols.clang-modules-cache-path "/home/mgorny/git/llvm-project/build/lldb-test-build.noindex/module-cache-lldb"
8: (lldb) settings set target.auto-apply-fixits false
9: (lldb) settings set target.inherit-tcc true
10: (lldb) settings set target.detach-on-error false
11: (lldb) target create "/home/mgorny/git/llvm-project/build/bin/lli"
12: Current executable set to '/home/mgorny/git/llvm-project/build/bin/lli' (x86_64).
13: (lldb) settings set plugin.jit-loader.gdb.enable on
14: (lldb) b jitbp
15: Breakpoint 1: no locations (pending).
16: WARNING: Unable to resolve breakpoint to any actual locations.
17: (lldb) run --jit-linker=jitlink /home/mgorny/git/llvm-project/build/tools/lldb/test/Shell/Breakpoint/Output/jit-loader_jitlink_elf.test.tmp.ll
check:13'0 X error: no match found
18: Process 536387 exited with status = 0 (0x00000000)
check:13'0 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
check:13'1 ? possible intended match
19: Process 536387 launched: '/home/mgorny/git/llvm-project/build/bin/lli' (x86_64)
check:13'0 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
>>>>>>
--
********************
********************
Failed Tests (2):
lldb-shell :: Breakpoint/jit-loader_jitlink_elf.test
lldb-shell :: Breakpoint/jit-loader_rtdyld_elf.test
Ok, it seems that either I was wrong and it isn't recent at all, or it happened because of some independent changes. I've tried bisecting back to Aug 2021 and it still happened.
If this really fails on Linux then it's a bug. I am not sure about the state of ORC/JITLink on FreeBSD, but I'd guess that it should work, because I don't see them opting out in the LLVM test configs (though please note that most of them are no execution tests): https://github.com/llvm/llvm-project/blob/main/llvm/test/ExecutionEngine/lit.local.cfg https://github.com/llvm/llvm-project/blob/main/llvm/test/ExecutionEngine/OrcLazy/lit.local.cfg
Well, I'm guessing they really dislike something about my setup then… I guess it's either how I'm configuring/building LLVM or something about my hardware (FreeBSD is running on qemu with KVM enabled).
My cmake args are:
CC=clang CXX=clang++ cmake ../llvm -G Ninja -DBUILD_SHARED_LIBS=ON -DCMAKE_BUILD_TYPE=MinSizeRel -DLLVM_TARGETS_TO_BUILD=all -DLLVM_ENABLE_ASSERTIONS=ON -DLLVM_CCACHE_BUILD=ON -DLLVM_ENABLE_PROJECTS='llvm;clang;lldb' -DPython3_EXECUTABLE=/usr/bin/python3.10
I don't have a FreeBSD system at hand to test it quickly, but if it fails for you it might happen to other people as well and it would be good to figure out what the issue is. Can you check if these two tests are failing as well? https://github.com/llvm/llvm-project/blob/release/15.x/llvm/test/ExecutionEngine/OrcLazy/debug-objects-elf-minimal.ll https://github.com/llvm/llvm-project/blob/release/15.x/llvm/test/ExecutionEngine/OrcLazy/debug-descriptor-elf-minimal.ll
They use a little extension in lli that I added back then to dump debug descriptor and in-memory debug objects. This is what lldb should be able to pick up and process in order to set the JIT breakpoints. Do you get some reasonable output here?
I don't have a FreeBSD system at hand to test it quickly [...].
Well, it fails for me on Linux as well. If you could try building with the CMake args I pasted above and see if you can reproduce it, that'd help us rule that one possibility out.
Can you check if these two tests are failing as well? https://github.com/llvm/llvm-project/blob/release/15.x/llvm/test/ExecutionEngine/OrcLazy/debug-objects-elf-minimal.ll https://github.com/llvm/llvm-project/blob/release/15.x/llvm/test/ExecutionEngine/OrcLazy/debug-descriptor-elf-minimal.ll
I'm on git main currently (65d83ba34378b8e740c5203fe46a9c50d2aeb862) and they are passing here:
PASS: LLVM :: ExecutionEngine/OrcLazy/debug-descriptor-elf-minimal.ll (4 of 23)
PASS: LLVM :: ExecutionEngine/OrcLazy/debug-objects-elf-minimal.ll (21 of 23)
(and the two Breakpoint/jit-loader tests are still failing)
They use a little extension in
llithat I added back then to dump debug descriptor and in-memory debug objects. This is whatlldbshould be able to pick up and process in order to set the JIT breakpoints. Do you get some reasonable output here?
If you mean:
./bin/lli --jit-linker=jitlink --generate=__dump_jit_debug_objects /home/mgorny/git/llvm-project/build/tools/lldb/test/Shell/Breakpoint/Output/jit-loader_jitlink_elf.test.tmp.ll
Then I get no output.
@mgorny So, I was able to reproduce the exact same error with your checkout and build config. It happened, because I missed to build lldb-server first. Building it fixes the issue for me on Linux. Can you double-check on your system? Thanks
check-lldb didn't reproduce the issue at all, because it has an input dependency to lldb-server. I guess that explains why this doesn't show up on the bots?
I'm afraid that's not it. It fails through the usual check-lldb for me consistently, and I'm sure I have an up-to-date lldb-server(after all, this is one of the things I'm working on all the time).
It must be specific to FreeSBD then. Bots agree that the Linux build is fine: https://lab.llvm.org/buildbot/#/builders/68/builds/38012
PASS: lldb-shell :: Breakpoint/jit-loader_rtdyld_elf.test (628 of 2278)
PASS: lldb-shell :: Breakpoint/jit-loader_jitlink_elf.test (1063 of 2278)
If you want to investigate this further, it might be worth starting here and see if you find anything helpful:
$ bin/lldb -- bin/lli
(lldb) target create "bin/lli"
Current executable set to '/home/ez/Develop/llvm-project/build/bin/lli' (x86_64).
(lldb) b jitbp
Breakpoint 1: no locations (pending).
WARNING: Unable to resolve breakpoint to any actual locations.
(lldb) run tools/lldb/test/Shell/Breakpoint/Output/jit-loader_jitlink_elf.test.tmp.ll
Process 188215 launched: '/home/ez/Develop/llvm-project/build/bin/lli' (x86_64)
1 location added to breakpoint 1
Process 188215 stopped
* thread #1, name = 'lli', stop reason = breakpoint 1.1
frame #0: 0x00007ffff7ffb004 JIT(0x4f46570)`jitbp() at jitbp.cpp:1:15
-> 1 int jitbp() { return 0; }
2 int main() { return jitbp(); }
It happens just the same on my Gentoo Linux system.
(lldb) target create "bin/lli"
Current executable set to '/home/mgorny/git/llvm-project/build/bin/lli' (x86_64).
(lldb) b jitbp
Breakpoint 1: no locations (pending).
WARNING: Unable to resolve breakpoint to any actual locations.
(lldb) run tools/lldb/test/Shell/Breakpoint/Output/jit-loader_jitlink_elf.test.tmp.ll
Process 255015 launched: '/home/mgorny/git/llvm-project/build/bin/lli' (x86_64)
Process 255015 exited with status = 0 (0x00000000)
When you enable event logging for the JIT category, what's different in your output?
(lldb) log enable lldb jit
(lldb) run tools/lldb/test/Shell/Breakpoint/Output/jit-loader_jitlink_elf.test.tmp.ll
lldb JITLoaderGDB::SetJITBreakpoint looking for JIT register hook
lldb JITLoaderGDB::SetJITBreakpoint looking for JIT register hook
lldb JITLoaderGDB::SetJITBreakpoint looking for JIT register hook
lldb JITLoaderGDB::SetJITBreakpoint setting JIT breakpoint
Process 198664 launched: '/home/ez/Develop/llvm-project/build/bin/lli' (x86_64)
intern-state JITLoaderGDB::JITDebugBreakpointHit hit JIT breakpoint
intern-state JITLoaderGDB::ReadJITDescriptorImpl registering JIT entry at 0x4f46570 (3408 bytes)
1 location added to breakpoint 1
Process 198664 stopped
* thread #1, name = 'lli', stop reason = breakpoint 1.1
frame #0: 0x00007ffff7ffb004 JIT(0x4f46570)`jitbp() at jitbp.cpp:1:15
-> 1 int jitbp() { return 0; }
2 int main() { return jitbp(); }
(lldb) target create "bin/lli"
Current executable set to '/home/mgorny/git/llvm-project/build/bin/lli' (x86_64).
(lldb) log enable lldb jit
(lldb) run tools/lldb/test/Shell/Breakpoint/Output/jit-loader_jitlink_elf.test.tmp.ll
lldb JITLoaderGDB::SetJITBreakpoint looking for JIT register hook
lldb JITLoaderGDB::SetJITBreakpoint looking for JIT register hook
lldb JITLoaderGDB::SetJITBreakpoint looking for JIT register hook
lldb JITLoaderGDB::SetJITBreakpoint looking for JIT register hook
Process 182588 launched: '/home/mgorny/git/llvm-project/build/bin/lli' (x86_64)
intern-state JITLoaderGDB::SetJITBreakpoint looking for JIT register hook
Process 182588 exited with status = 0 (0x00000000)
Alright, then LLDB doesn't find the GDB JIT Interface symbols __jit_debug_register_code and __jit_debug_register_descriptor. Make sure lli exposes them:
$ llvm-nm bin/lli | grep __jit_debug
0000000004df13c8 D __jit_debug_descriptor
00000000033440d0 T __jit_debug_register_code
..and debug from here: https://github.com/llvm/llvm-project/blob/main/lldb/source/Plugins/JITLoader/GDB/JITLoaderGDB.cpp#L191
BTW JITed code debugging was broken for a while across platforms until https://github.com/llvm/llvm-project/issues/35557
Alright, then LLDB doesn't find the GDB JIT Interface symbols
__jit_debug_register_codeand__jit_debug_register_descriptor. Make surelliexposes them:$ llvm-nm bin/lli | grep __jit_debug 0000000004df13c8 D __jit_debug_descriptor 00000000033440d0 T __jit_debug_register_code
Yep, they're not there. Since it links to shared libraries, they're in:
lib/libLLVMExecutionEngine.so
lib/libLLVMOrcTargetProcess.so
Yep, they're not there. Since it links to shared libraries, they're in:
lib/libLLVMExecutionEngine.so lib/libLLVMOrcTargetProcess.so
Right! It turns out that I had started testing with -DBUILD_SHARED_LIBS=ON and accidentally fell back to my regular (non-shared-libs build) later. Sorry for that.
I never tested, whether or not the JIT debug breakpoint (that LLDB sets implicitly in __jit_debug_register_code) gets set correctly if it's located in a shared library. I assumed it behaves like any other breakpoint that can resolve extra locations when loading extra shared libraries. Just checked with lldb-10 and it doesn't work there either.
Curious enough, it seems to work fine when using dylib rather than split shared libraries, even though the symbol is in .so rather than the executable
$ nm -B lib/libLLVM-16git.so | grep __jit
0000000006de69e8 D __jit_debug_descriptor
00000000024aac84 T __jit_debug_register_code
I wonder if these U entries could be the problem:
$ nm -B lib/libLLVMExecutionEngine.so | grep __jit
U __jit_debug_descriptor
U __jit_debug_register_code
$ nm -B lib/libLLVMOrcTargetProcess.so | grep __jit
000000000002b5d8 D __jit_debug_descriptor
000000000000f80a T __jit_debug_register_code
I wonder if these U entries could be the problem:
They might be. It would be worth checking what does the
addr_t jit_addr = GetSymbolAddress(
module_list, ConstString("__jit_debug_register_code"), eSymbolTypeAny);
line return for the module containing the undefined symbol...
