Missing external sources when debugging with dlv

[prestonvanloon]

To reproduce:

1. Have some bazel project with an external library provided by go_repository.
2. Build with bazel build -c dbg //program:main
3. Execute dlv exec bazel-bin/program/linux_amd64_debug/main
4. Set a breakpoint in the external library b github.com/abc/lib.(*Foo).Bar
5. Enter c to continue the execution until the breakpoint is hit
6. List the current source code with ls and see Command failed: open external/github.com/abc/lib/foo.go: no such file or directory

Is there any workaround besides not using bazel to build the debug binary?

[jayconrod]

It doesn't seem like we're stripping directory prefixes correctly at the moment. I think there are basically four cases here:

• Static files in the main workspace: these should work.
• Generated files in the main workspace: these should work.
• Files in the standard library. These do not work because the paths contain the string GOROOT, e.g., ./GOROOT/src/internal/cpu/cpu.go.
• Files in external repositories. These do not work because their path is relative to the execroot, not the workspace root, e.g., ./external/com_github_pkg_errors/stack.go

[devshorts]

Is there any progress on this? This is pretty frustrating

[dragonsinth]

I noticed the whole external/com_github_pkg_errors/stack.go source path formulation as we started building binaries in bazel and noticed that the stack traces differ now. Is that the intended source path for external go files over something like github.com/pkg/errors/stack.go?

[danny-skydio]

Something that works for me:

1. Build with bazel build -c dbg //program:main
2. Execute dlv exec bazel-bin/program/linux_amd64_debug/main
3. Map external/ path config substitute-path external/ /path/to/workspace-name/bazel-workspace-name/external/
4. Map GOROOT/ path config substitute-path external/ /path/to/workspace-name/bazel-workspace-name/external/go_sdk/

I put in an absolute path, but that's just out of convenience.

[sluongng]

Interesting 🤔

Perhaps we can document somewhere that user can invoke Delve with a special config file where substitute-path is set to absolute path of external and GOROOT 🤔

[stagnation]

Interesting indeed :) I finally wrote up how I debug go programs (and the same techniques for c/c++). As I work in multiple projects at the same time, I prefer to use the initialization commands over the config file. I guess we could find a remapping scheme that takes the project into account - but I did not dig into the debug symbols to find how to map them, and work with multiple go projects.

If you want I could write this up as an official document too: https://meroton.com/docs/tips/debug-bazel-go-projects/

[sluongng]

@stagnation I read your first draft as well as the latest draft. Very cool find!

I wonder if we could create something like go_debug_binary where the executable orchestrates executing dlv and the dependent binary automagically 🤔, including correct source mapping and a transition to -c debug.

If that's possible, then we could make go_binary be a macro, exposing an extra name + ".debug" target for folks to develop easier.

[stagnation]

Thank you :)

Hmm, I guess that would work, I really appreciate the pain point and hear of it frequently, but also want to minimize auxiliary targets through macros. As they tend to bloat queries the big picture of the dependency tree gets lost (without fancy filters). I do not know which is most important for me, so do not take it as an objection, just pointing out the tension I feel.

In my eyes the best would be to drive debugging through --run_under=@rules_go//:debugger which could probably find the remappings for us, but the TTY issue is very annoying and has troubled me since I first started using Bazel. The cat trick generally works - but then we are essentially in the land of a bazel-wrapper with its own costs. https://github.com/bazelbuild/bazel/issues/11371 https://github.com/bazelbuild/bazel/issues/13934

To complicate things further, I would also like to debug with rr, whose recording works well with run_under, as the debugger is launched post-hoc. But rr requires cgo to my understanding, have not explored go debugging with rr sufficiently to report yet.

[sluongng]

I think if we tag the go_debug_binary with "manual", it would help filter things out a bit?

The downside of using --run_under is that you cannot apply a transition to debug configuration automatically.

Im not familiar with using rr on go binaries to comment on the feasibility. I suspect such a use case is relatively rarer vs dlv

[fmeum]

Looks like we can do quite a bit here to make the experience better, thanks @stagnation for the detailed ideas!

Note that go_binary already transitions itself, so we could add more settings to it without adding new targets. Alternatively, we could make -c dbg the switch for this behavior.

I will try to delve into the related Bazel issues and see whether we can make some progress there.

[sluongng]

I think if we gona have to invoke dlv with this

$ dlv exec --init <(echo '
    config substitute-path external/ "$OUTPUT_BASE"/external
    config substitute-path GOROOT/   "$OUTPUT_BASE"/external/go-sdk
    config substitute-path ""        "$OUTPUT_BASE"/
    print "You can also set break points and automatically start the program."
    break main.main
    continue
') $(location //some/go_binary) -- -h

then we might as well construct this bash script with Bazel and make it a sh_binary with 2 data dependencies: dlv and the target go_binary