Breakpoints in plugins are not working if set before plugin is loaded

[akamensky]

trafficstars

Please answer the following before submitting your issue:

1. Version: 1.2.0 Build: dd4fd5dc9cf5d45092f97f2ba376f3e69d7a5a8e
2. What version of Go are you using? go version go1.12.7 linux/amd64
3. What operating system and processor architecture are you using? linux x86_64
4. What did you do?

Attempt to debug plugin from Goland. Using i.e. https://github.com/vladimirvivien/go-plugin-example.git (compile all plugins with -gcflags='all=-N -l'). Place breakpoints inside functions of plugin and inside main application functions.

5. What did you expect to see?

Breakpoints of main application work. Breakpoints inside plugins work.

6. What did you see instead?

Breakpoints of main application work. Plugin breakpoints are ignored.

[derekparker]

Is there an error or any odd output when you put a breakpoint in the plugin code?

[akamensky]

Is there an error or any odd output when you put a breakpoint in the plugin code?

No errors, the breakpoints are just skipped as if non-existent

[aarzilli]

I'd like to see output of --log --log-output=rpc. I'm guessing you are setting all breakpoints before starting the program, the ones on plugins fail (because the plugins aren't loaded when the program starts) and goland isn't telling you. We should have delayed breakpoints, but we don't.

[akamensky]

@aarzilli you are right I am setting breakpoints before starting the application.

Are the arguments you mention for dlv?

[aarzilli]

Yes. IIRC there's an option to get those logs from GoLand, I think maybe somewhere under the help menu?

[akamensky]

I will get the log later (did not find it this option anywhere in GoLand btw). Meanwhile, it appears that even after the module loaded I am unable to set breakpoints via IDE, I've set a breakpoint right before call to Greet() and when I hit it, navigate to the source file and set breakpoint there, this normally works with non-plugin code, but here it appears as disabled/inactive breakpoint:

BTW, the second screenshot is from Mac, so not sure if plugin debugging is even supposed to work there, I can only try again on Linux later, where I can also try to get logs from debugger.

[akamensky]

To get this log from IDE I had to make a wrapped bash in place of dlv binary that calls actual dlv binary with extra arguments :heavy_sigh:

Found that in both cases whether I try to create breakpoint before starting application or wait until plugin is loaded it shows same exact error:

2019-08-13T22:49:35+08:00 debug layer=rpc <- RPCServer.CreateBreakpoint(rpc2.CreateBreakpointIn{"Breakpoint":{"id":0,"name":"","addr":0,"file":"/Users/alexey.kamenskiy/GolandProjects/go-plugin-example/eng/greeter.go","line":8,"Cond":"","continue":false,"traceReturn":false,"goroutine":false,"stacktrace":0,"LoadArgs":null,"LoadLocals":null,"hitCount":null,"totalHitCount":0}})
2019-08-13T22:49:35+08:00 debug layer=rpc -> *rpc2.CreateBreakpointOut{"Breakpoint":{"id":0,"name":"","addr":0,"file":"","line":0,"Cond":"","continue":false,"traceReturn":false,"goroutine":false,"stacktrace":0,"LoadArgs":null,"LoadLocals":null,"hitCount":null,"totalHitCount":0}} error: "could not find file /Users/alexey.kamenskiy/GolandProjects/go-plugin-example/eng/greeter.go"

However the file is most definitely there:

$ ls -la /Users/alexey.kamenskiy/GolandProjects/go-plugin-example/eng/greeter.go
-rw-r--r--  1 alexey.kamenskiy  staff  145 Aug 13 22:26 /Users/alexey.kamenskiy/GolandProjects/go-plugin-example/eng/greeter.go
$

Also on Darwin, when i create breakpoint before running application, it automagically pops an Assembly file in editor "paused" on a RET instruction (same does not happen on Linux):

[aarzilli]

Plugin debugging doesn't work on macOS

[akamensky]

@aarzilli fair enough, I’ve sorted of expected this on Mac since it seems support for it lagging on all fronts.

I’ll re-test tomorrow with logging on Linux and follow up here.

[akamensky]

Tested on Linux and it appears that if I set breakpoints before start application, then it skips them with same error: could not find file /path/to/file.go. If I, however, create a breakpoint before calling plugin method and once it hits that breakpoint set one inside plugin -- this new breakpoint works as expected.

There is one major issue with this though -- if I restart process the breakpoint inside plugin is no longer working.

Now sure if this is possible to detect from delve, but I would imagine "re-evaluating" breakpoints when detecting plugins being loaded should fix this issue? (re-evaluating here means check if all attempted breakpoints were successfully set, if not all, then check if any of unsuccessful are referring to loaded plugin, if they are -- set them at this point of time)

[akamensky]

Also I think this is independent from #1628 which refers to specifically Darwin platform where debugging does not work at all.

[nd]

Filed an issue for GoLand: https://youtrack.jetbrains.com/issue/GO-7957. Maybe we can fix it without changes in Delve. E.g. to set a invisible breakpoint inside the plugin.Open and re-sent breakpoints for which Delve din't find files.

[akamensky]

@nd I don't think this has anything to do with GoLand itself. From how Delve sets breakpoints the logic GoLand follows looks fine. GoLand would not know about those details, it passes breakpoint information to Delve, which in turn should be able to handle it. Also GoLand is not the only IDE, but Delve is basically only debugger that works for Go.

[FloThinksPi]

Hmm would it generally be possible for delve to "remember" breakpoints it has no reference to (executable does not contain debug information for file /somefile.go) and re-evaluates these breakpoints if/after a plugin load has occurred ?

[aarzilli]

Yes but there is no existing code for either: (a) deferring a breakpoint creation, (b) catching a plugin load as soon as it happens

[pepjo]

I also hit this issue when trying to debug a plugin, but in my case, it happened in vs code.

@aarzilli I would be interested in implementing the needed features if there is an agreed path.

[aarzilli]

It would probably have to be something like this:

• figure out where to put a breakpoint so that it gets hit after a plugin has been loaded (it needs to work on all supported os/arch combinations)
• add a callback function to proc.Breakpoint and call it when the breakpoint is detected in Continue
• change service/debugger.Debugger to use these two mechanisms to implement deferred breakpoints
• add an option to the CreateBreakpoint API argument to create a deferred breakpoint
• change the break command to ask if the user wants to create a deferred breakpoint when normal breakpoint creation doesn't work

[pepjo]

Thanks for outlining the tasks. However, It has become apparent that I will not have much time to work on this the following months, hopefully someone else can give it a try.

[YIDWang]

is there any documentation available or tips you can provide for debugging plugin code in mac? plugin still can't debug on mac。

[FloThinksPi]

@YIDWang

is there any documentation available or tips you can provide for debugging plugin code in mac? plugin still can't debug on mac。

See https://github.com/golang/go/issues/25841, not possible on mac atm afaik.