Some libraries abruptly stopped being debuggable after previously working

[noodlecollie]

OS: Kubuntu 23.04 VSCode version: (Codium) 1.80.0, Release: 23188 CodeLLDB version: v1.9.2 (version command reports lldb version 16.0.0-custom) Compiler: GCC (Ubuntu 12.3.0-1ubuntu1~23.04) 12.3.0 Debuggee: Executable with shared libraries, x86_64-linux-gnu

The other day, between ordinary builds when iterating on development, breakpoints stopped functioning in just two of my shared libraries. I initially made a StackOverflow post about the issue (https://stackoverflow.com/questions/76948118/lldb-cannot-find-breakpoint-locations-in-library-with-debug-info-present), and have since managed to establish that these libraries are not included in the output of image list. I have no idea why this, but I suspect it's something to do with the debugger.

Both of the libraries do include debug info as confirmed by running file, and were debuggable without issue until whatever it is happened here.

I have tried the following to resolve this issue, without success:

• Deleting my build folder and doing a clean rebuild.
• Reverting to previous Git commits/other Git branches and rebuilding.
• Restarting VS Codium.
• Restarting my machine.
• Reverting to previous versions of the CodeLLDB and CMake Tools extensions.
• Updating the packages on my system.

A verbose log of me beginning a debugging session and locating two different symbols is below. The commands I use to test are:

• image lookup -f cl_cmds.c -l 26 - this is a symbol in one of the libraries that does get correctly loaded.
• image lookup -f hud_crosshair.cpp -l 50 - this is a symbol in a different library, that I was trying to debug at the time, which is no longer found.

Verbose log

Initial debug configuration: {
  type: 'lldb',
  request: 'launch',
  name: 'Debug',
  program: '${workspaceFolder}/build/install/nightfire-open/bond',
  args: [ '-dev', '5', '+map', 'debug_target' ],
  env: {
    LD_LIBRARY_PATH: '${workspaceFolder}/build/install/nightfire-open'
  },
  cwd: '${workspaceFolder}',
  __configurationTarget: 6
}
Resolved debug configuration: {
  type: 'lldb',
  request: 'launch',
  name: 'Debug',
  program: '${workspaceFolder}/build/install/nightfire-open/bond',
  args: [ '-dev', '5', '+map', 'debug_target' ],
  env: {
    LD_LIBRARY_PATH: '${workspaceFolder}/build/install/nightfire-open'
  },
  cwd: '${workspaceFolder}',
  __configurationTarget: 6,
  sourceLanguages: [ 'C', 'C++' ],
  relativePathBase: '/home/vesper/Documents/nightfire-open',
  _adapterSettings: {
    displayFormat: 'auto',
    showDisassembly: 'auto',
    dereferencePointers: true,
    suppressMissingSourceFiles: true,
    evaluationTimeout: 5,
    consoleMode: 'commands',
    sourceLanguages: null,
    terminalPromptClear: null,
    evaluateForHovers: true,
    commandCompletions: true,
    reproducer: false
  }
}
liblldb: /home/vesper/.vscode-oss/extensions/vadimcn.vscode-lldb-1.9.2-universal/lldb/lib/liblldb.so
environment: {}
settings: {
  evaluateForHovers: true,
  commandCompletions: true,
  sourceLanguages: [ 'C', 'C++' ]
}
[DEBUG codelldb] Connecting to 127.0.0.1:34029
[DEBUG codelldb] New debug session
[DEBUG codelldb::dap_codec] --> {"command":"initialize","arguments":{"clientID":"vscode","clientName":"VSCodium","adapterID":"lldb","pathFormat":"path","linesStartAt1":true,"columnsStartAt1":true,"supportsVariableType":true,"supportsVariablePaging":true,"supportsRunInTerminalRequest":true,"locale":"en","supportsProgressReporting":true,"supportsInvalidatedEvent":true,"supportsMemoryReferences":true,"supportsArgsCanBeInterpretedByShell":true,"supportsMemoryEvent":true,"supportsStartDebuggingRequest":true},"type":"request","seq":1}
[DEBUG codelldb::dap_codec]  {"command":"launch","arguments":{"type":"lldb","request":"launch","name":"Debug","program":"/home/vesper/Documents/nightfire-open/build/install/nightfire-open/bond","args":["-dev","5","+map","debug_target"],"env":{"LD_LIBRARY_PATH":"/home/vesper/Documents/nightfire-open/build/install/nightfire-open"},"cwd":"/home/vesper/Documents/nightfire-open","__configurationTarget":6,"sourceLanguages":["C","C++"],"relativePathBase":"/home/vesper/Documents/nightfire-open","_adapterSettings":{"displayFormat":"auto","showDisassembly":"auto","dereferencePointers":true,"suppressMissingSourceFiles":true,"evaluationTimeout":5,"consoleMode":"commands","sourceLanguages":null,"terminalPromptClear":null,"evaluateForHovers":true,"commandCompletions":true,"reproducer":false},"__sessionId":"f398e48d-ffd0-4875-bfa9-a048f7570268"},"type":"request","seq":2}
[DEBUG codelldb::dap_codec]  {"command":"setFunctionBreakpoints","arguments":{"breakpoints":[]},"type":"request","seq":3}
[DEBUG codelldb::dap_codec]  {"command":"setDataBreakpoints","arguments":{"breakpoints":[]},"type":"request","seq":4}
[DEBUG codelldb::dap_codec]  {"command":"setInstructionBreakpoints","arguments":{"breakpoints":[]},"type":"request","seq":5}
[DEBUG codelldb::dap_codec]  {"type":"response","seq":6,"command":"runInTerminal","request_seq":5,"success":true,"body":{"shellProcessId":3693555}}
[DEBUG codelldb::dap_codec] --> {"command":"configurationDone","type":"request","seq":7}
[DEBUG codelldb::dap_codec]  {"command":"threads","type":"request","seq":8}
[DEBUG codelldb::dap_codec]  {"command":"evaluate","arguments":{"expression":"image lookup -f cl_cmds.c -l 26","context":"repl"},"type":"request","seq":9}
[DEBUG codelldb::dap_codec]  SuccessFinishResult, Error:  Success
    Output Message:
    2 matches found in cl_cmds.c:26 in /home/vesper/Documents/nightfire-open/build/install/nightfire-open/libxash.so:
            Address: libxash.so[0x00000000000d850b] (libxash.so.PT_LOAD[1]..text + 784731)
            Summary: libxash.so`CL_PlayVideo_f at cl_cmds.c:27:1        Address: libxash.so[0x00000000000d851a] (libxash.so.PT_LOAD[1]..text + 784746)
            Summary: libxash.so`CL_PlayVideo_f + 15 at cl_cmds.c:27:1
[DEBUG codelldb::dap_codec] <-- {"seq":111,"type":"response","request_seq":9,"success":true,"command":"evaluate","body":{"result":"","variablesReference":0}}
[DEBUG codelldb::dap_codec] --> {"command":"evaluate","arguments":{"expression":"image lookup -f hud_crosshair.cpp -l 50","context":"repl"},"type":"request","seq":10}
[DEBUG codelldb::debug_session::variables] image lookup -f hud_crosshair.cpp -l 50 -> Failed, Error:  Fail
[ERROR codelldb::debug_session]
[DEBUG codelldb::dap_codec] <-- {"seq":112,"type":"response","request_seq":10,"success":false,"command":"","message":"","show_user":true}
[DEBUG codelldb::dap_codec] --> {"command":"disconnect","arguments":{"restart":false,"terminateDebuggee":true},"type":"request","seq":11}
[DEBUG codelldb::debug_session] Debug event: 0x5568a2fe31d0 Event: broadcaster = 0x5568a2e8d6a8 (lldb.process), type = 0x00000001 (state-changed), data = { process = 0x5568a2e8d670 (pid = 531832), state = exited}
[DEBUG codelldb::dap_codec] <-- {"seq":113,"type":"response","request_seq":11,"success":true,"command":"disconnect"}
[DEBUG codelldb::dap_codec] <-- {"seq":114,"type":"event","event":"output","body":{"category":"console","output":"Process exited with code 9.\n"}}
[DEBUG codelldb::dap_codec] <-- {"seq":115,"type":"event","event":"exited","body":{"exitCode":9}}
[DEBUG codelldb::dap_codec] <-- {"seq":116,"type":"event","event":"terminated","body":{}}
[DEBUG codelldb::dap_session] Client has disconnected
[DEBUG codelldb::debug_session] End of the requests stream
[DEBUG codelldb::debug_session] DebugSession::drop()
[DEBUG codelldb] End of the debug session
[DEBUG codelldb] Exiting
Debug adapter exit code=0 (0x0), signal=null.

[vadimcn]

Why do you think this is a debugger issue?

[noodlecollie]

I'd exhausted a number of other options. I've tried rebuilding, building old commits where the breakpoints definitely used to work (they now do not), even reinstalling my entire system. If possible, I'd like to at least try and get some information from the debugger about why it can't load these particular libraries, because currently I have nothing more to go on.

Recently, someone else commented on my SO post saying "I think I have the same problem, but only for lldb-vscode on llvm 16.0.6", so I guess I'm not the only one having this problem.

[vadimcn]

Still doesn't mean it's a debugger problem. Have you made sure your symbol is used anywhere? Does it actually get called? Is your library on the loaded modules list? Is the debug info for you source file actually there? (you can use e.g. dwarfdump or llvm-dwarfdump to dump debug info in text form, then search it for file names and function names).

[noodlecollie]

Have you made sure your symbol is used anywhere? Does it actually get called?

Absolutely. For context, breakpoints used to work 100% fine, until one day when I was developing something and did an incremental build, they stopped working. I reverted the change and they still didn't work, and nothing I've done since has fixed them, even reverting to far older commits.

Is your library on the loaded modules list?

It is not, even though the application is definitely using it. All the other libraries used by the application are listed.

Is the debug info for you source file actually there?

Yes. dwarfdump lists everything I would expect it to.