add support for debugging child process enabled by delve backend

[Lslightly]

Problem demo: The demo repository is Lslightly/delve-dbg-child-proc, where I can debug child process in delve. But I can't debug child process in vscode-go.

Although most go projects are organized as package and support of single process with multiple goroutines is enough, there are projects like golang/go where the cmd/go can invoke cmd/compile tool as the child process. The command arguments passed to cmd/compile tool is complicated. It will be convenient to debug cmd/compile through debugging go build -a . command instead of debugging go tool compile ... ... ... command. So demand of multiple process debugging feature exist. Besides, this scenario(use case) is different from the client/server architecture which may run the process for a long time and you can connect to the delve server at any time(or make the child process sleep for some seconds).

Related issues and blogs:

• https://github.com/go-delve/delve/issues/2551
• https://github.com/go-delve/delve/pull/3937
• https://github.com/golang/vscode-go/discussions/3704
• https://www.dolthub.com/blog/2023-05-25-debugging-multiple-golang-processes/

Describe the solution you'd like A clear and concise description of what you want to happen.

1. Enable target follow-exec -on by default or provide an option to enable child process debug. (--init option of delve is disabled when combining with --headless option)
2. ~~Allow setting breakpoint whose location is not found yet but will be found in child process.~~ There are some mistakes. See the comment
   • ~~This needs support in delve to answer yes when the breakpoint is not found to set a suspended breakpoint(in delve but not in vscode-go. vscode-go can always set the breakpoint on since there is no side effect).~~
   • ~~The source code location of yesno decision is pkg/terminal/command.go#L1868~~
3. Reuse the Call Stack for debugging child process. (low priority. Since attach remote and substitutePath(I haven't succeeded in using this feature yet) can be used to debug child process in different vscode window.

Describe alternatives you've considered A clear and concise description of any alternative solutions or features you've considered.

Additional context Add any other context or screenshots about the feature request here.

I will upload a demo video later.

[Lslightly]

2. Allow setting breakpoint whose location is not found yet but will be found in child process.
   • This needs support in delve to answer yes when the breakpoint is not found to set a suspended breakpoint(in delve but not in vscode-go. vscode-go can always set the breakpoint on since there is no side effect).
   • The source code location of yesno decision is pkg/terminal/command.go#L1868

Sorry, I made some mistakes.

1. vscode-go interacts with service/debugger/debugger.go CreateBreakpoint instead of yesno in pkg/terminal/command.go, which is the terminal client/frontend of delve.

2. vscode-go successfully set breakpoints whose location is not found through CreateBreakpoint

Delve provides RPCServer.FollowExec which is equal to target follow-exec -on/-off command. However, now vscode-go does not use this API.

The only thing left is to make calling RPCServer.FollowExec possible.

[firelizzard18]

@Lslightly Does delve support this? When delve is running in DAP mode, it would need to send a StartDebugging request back to VSCode and I don't see any evidence that it is doing so. I am interested in this and have a project that it would be helpful for (so maybe I'll be motivated enough to do it myself) but I want to be sure I know what work actually needs to be done.

[Lslightly]

Does delve support this?

If I understand correctly, "this" refers to child process debugging. The answer is yes. In terminal mode of delve, using target follow-exec -on does the work. For vscode-go, you can follow instructions in https://github.com/golang/vscode-go/issues/3713#issuecomment-2718370472.

When delve is running in DAP mode, it would need to send a StartDebugging request back to VSCode and I don't see any evidence that it is doing so.

I'm not familiar with DAP. But StartDebugging request should be received by vscode-go if my memory is not wrong?

I want to be sure I know what work actually needs to be done

See https://github.com/go-delve/delve/pull/3948. It may help.

If you are working on that, please tell me the progress as I'm also interested in it but I don't have aerial view of DAP protocol and vscode-go goDebug logic and I don't have time these days.

[firelizzard18]

The Debug Adapter Protocol is a common protocol for editors to talk with debuggers. Basically all modern vscode debuggers are DAP debuggers. The StartDebugging request is the tool DAP provides to allow a debugger like delve to notify an editor like vscode that there is a child process or some other scenario where a new debug session should be started.

Originally, vscode-go implemented a wrapper that translated between DAP and delve's own RPC/API. This is the legacy debug adapter. Since then, delve has added native support so it can talk directly to vscode and vscode-go is only responsible for launching the process. This is the dlv-dap debug adapter. Development has halted on the legacy debug adapter and it will be removed eventually. Given that the modern adapter (dlv-dap) talks directly to vscode without vscode-go's involvement, must debugger features must be implemented directly in delve and cannot be implemented in vscode-go.

Does delve support this?

I mean, does delve support target follow-exec -on when launched with dlv dap ...?

I'm not familiar with DAP. But StartDebugging request should be received by vscode-go if my memory is not wrong?

Small correction, StartDebugging is received by VSCode, not vscode-go - vscode-go is only responsible for launching the debug session, it doesn't really participate after that.

My point is, I don't see any evidence that delve ever sends that request. AFAIK for child process debugging to work, we need two things: we need a way to enable target follow-exec -on and we need delve to send the StartDebugging request. But I've searched through delve's source code and I don't see any evidence that it ever sends that request.

TL;DR: Enabling target follow-exec -on by default or with an option won't work because AFAIK delve does not support follow-exec when launched with dlv dap ....

[Lslightly]

Thanks for your detailed explanation! Sorry for the misleading title because it's not enabled yet.

I found that I misunderstood StartDebugging request because I was still thinking about my debugging experience in legacy debugger.

I check the StartDebugging request and indeed it does not appear in DAP service of dlv. Since StartDebugging request is a reverse request sent by debug adapter in dlv, I check how new target process is detected by dlv. The process is as follows:

1. in pkg/proc/native/proc_linux.go, procgrp.add is called to add new target process in process group in trapWaitInternal function.
2. trapWaitInternal will be called by Continue in pkg/proc/target_exec.go(There are some call chains omitted).

After creating new target, grp.pickCurrentTarget(target) called by Continue will pick one target process. The return state of s.runUntilStop(command, allowNextStateChange) which is called indirectly by onContinueRequest(many call chains omitted) can be different after switching different process. So this can be point for creating StartDebugging request?

[firelizzard18]

in pkg/proc/native/proc_linux.go, procgrp.add is called to add new target process in process group in trapWaitInternal function.

Makes sense.

The return state of s.runUntilStop(command, allowNextStateChange) which is called indirectly by onContinueRequest(many call chains omitted) can be different after switching different process. So this can be point for creating StartDebugging request?

You lost me there, I haven't dug that deeply into delve. I think what you're saying makes sense but whether I understand it doesn't really matter. Your PR https://github.com/go-delve/delve/pull/3948 seems like a good start. I think the next step is to create feature requests (issues) in delve for being able to set follow-exec via a command or flag or something, and another for implementing the StartDebugging request. I'm not sure what action aarzilli's comment about ProcessEvent notifications requires.

I know a decent amount about DAP having built a couple DAP debuggers myself, but I know very little about delve's internals.

[Lslightly]

Based on

1. firelizzard18/delve, which enables suspending breakpoints for plugin.Open.
2. commit d9991f, which enables unsuspending any suspended breakpoints when hitting them.
3. commit d52c26 in delve and commit 1a841f in vscode-go, which add follow-exec and follow-exec regex configuration options in the interface for LaunchRequest and AttachRequest(not tested yet).
4. commit 7b8a1d in delve, which support using dlv target list, dlv target switch(dangerous now), dlv target follow-exec [-on [regex]] [-off].

Although not tested thoroughly, vscode-go can now debug child process now! :tada:

Here are some demos:

vscode-go:

https://github.com/user-attachments/assets/76b67507-4d1a-4011-b1f4-1fd338ef6c4b

Debug Console:

https://github.com/user-attachments/assets/5fa12430-48a6-4d3a-8a82-c0bc836afbd9

dlv follow-exec -off:

https://github.com/user-attachments/assets/76cb5906-8f3d-4ea2-b218-d6bb58aaf0b3

dlv follow-exec -on:

https://github.com/user-attachments/assets/ffd88013-a48b-45b8-9a82-9701b0a2e495

Here are two examples: Plugin and Lslightly/delve-dbg-child-proc you can try.

There are several things to do yet:

1. testing for delve.
2. If child process is spawned, then suspended breakpoints can be verified immediately if possible.
3. delve sends StartDebugging request back to vscode.

[gopherbot]

Change https://go.dev/cl/696795 mentions this issue: extension: add followExec and followExecRegex options in launch request