delve
delve copied to clipboard
go-delve
Cannot debug go plugins on mac
- What version of Delve are you using (
dlv version)?
Delve Debugger
Version: 1.2.0
Build: cb658772972d74042ba75f006ac14023e552c7cd
- What version of Go are you using? (
go version)?
go version go1.12.6 darwin/amd64
- What operating system and processor architecture are you using?
darwin/amd64
- What did you do?
I'm trying to debug a simple go plugin from https://github.com/vladimirvivien/go-plugin-example.
I've build plugins using the command:
go build -gcflags "all=-N -l" -x -v -buildmode=plugin -o eng/eng.so eng/greeter.go
I'm debugging the code calling the plugin (https://github.com/vladimirvivien/go-plugin-example/blob/master/greeter.go).
Once debugger reaches the breakpoint on line 58, I'm adding a breakpoint at eng/greeter.go:8 and continue or trying to step into the call.
- What did you expect to see?
Adding a breakpoint inside the plugin succeeds. If I continue, debugger hits the breakpoint inside the plugin. Step into the call in plugin also works.
- What did you see instead?
Create breakpoint call fails with the "could not find file /Users/nd/go/src/github.com/vladimirvivien/go-plugin-example/eng/greeter.go" error. Step into plugin call also fails.
Debug log: https://gist.github.com/nd/feb71212f6ef9dcfbb33c4293e18f146
Debug works fine on linux.
Debugging plugin on mac is really needed for Golang developers. Hope delve supports it :)
I didn't try implementing this because it would be affected by https://github.com/golang/go/issues/25841 at least. I also vaguely remembering some versions of go not producing any dwarf at all for macOS. What's the output of readelf -h -S path_to_a_go_plugin.so?
It returns an error:
readelf: Error: Not an ELF file - it has the wrong magic bytes at the start
It returns an error:
Right, of course, because they aren't elfs. My bad. I guess the right one would be otool -l or something like that. Or gobjdump -h ... if it's installed.
otool -l output:
eng/eng.so:
Mach header
magic cputype cpusubtype caps filetype ncmds sizeofcmds flags
0xfeedfacf 16777223 3 0x00 6 13 1984 0x00100004
Load command 0
cmd LC_SEGMENT_64
cmdsize 392
segname __TEXT
vmaddr 0x0000000000000000
vmsize 0x00000000000dd000
fileoff 0
filesize 905216
maxprot 0x00000007
initprot 0x00000005
nsects 4
flags 0x0
Section
sectname __text
segname __TEXT
addr 0x0000000000002040
size 0x00000000000dabd0
offset 8256
align 2^4 (16)
reloff 0
nreloc 0
flags 0x80000400
reserved1 0
reserved2 0
Section
sectname __stubs
segname __TEXT
addr 0x00000000000dcc10
size 0x0000000000000156
offset 904208
align 2^1 (2)
reloff 0
nreloc 0
flags 0x80000408
reserved1 0 (index into indirect symbol table)
reserved2 6 (size of stubs)
Section
sectname __stub_helper
segname __TEXT
addr 0x00000000000dcd68
size 0x000000000000024a
offset 904552
align 2^2 (4)
reloff 0
nreloc 0
flags 0x80000400
reserved1 0
reserved2 0
Section
sectname __unwind_info
segname __TEXT
addr 0x00000000000dcfb4
size 0x0000000000000048
offset 905140
align 2^2 (4)
reloff 0
nreloc 0
flags 0x00000000
reserved1 0
reserved2 0
Load command 1
cmd LC_SEGMENT_64
cmdsize 1112
segname __DATA
vmaddr 0x00000000000dd000
vmsize 0x0000000000122000
fileoff 905216
filesize 1064960
maxprot 0x00000007
initprot 0x00000003
nsects 13
flags 0x0
Section
sectname __nl_symbol_ptr
segname __DATA
addr 0x00000000000dd000
size 0x0000000000000010
offset 905216
align 2^3 (8)
reloff 0
nreloc 0
flags 0x00000006
reserved1 57 (index into indirect symbol table)
reserved2 0
Section
sectname __got
segname __DATA
addr 0x00000000000dd010
size 0x0000000000001500
offset 905232
align 2^3 (8)
reloff 0
nreloc 0
flags 0x00000006
reserved1 59 (index into indirect symbol table)
reserved2 0
Section
sectname __la_symbol_ptr
segname __DATA
addr 0x00000000000de510
size 0x00000000000001c8
offset 910608
align 2^3 (8)
reloff 0
nreloc 0
flags 0x00000007
reserved1 731 (index into indirect symbol table)
reserved2 0
Section
sectname __mod_init_func
segname __DATA
addr 0x00000000000de6d8
size 0x0000000000000008
offset 911064
align 2^3 (8)
reloff 0
nreloc 0
flags 0x00000009
reserved1 0
reserved2 0
Section
sectname __rodata
segname __DATA
addr 0x00000000000de6e0
size 0x0000000000052d58
offset 911072
align 2^5 (32)
reloff 0
nreloc 0
flags 0x00000000
reserved1 0
reserved2 0
Section
sectname __typelink
segname __DATA
addr 0x0000000000131440
size 0x0000000000001434
offset 1250368
align 2^5 (32)
reloff 0
nreloc 0
flags 0x00000000
reserved1 0
reserved2 0
Section
sectname __itablink
segname __DATA
addr 0x0000000000132878
size 0x00000000000000a0
offset 1255544
align 2^3 (8)
reloff 0
nreloc 0
flags 0x00000000
reserved1 0
reserved2 0
Section
sectname __gosymtab
segname __DATA
addr 0x0000000000132918
size 0x0000000000000000
offset 1255704
align 2^0 (1)
reloff 0
nreloc 0
flags 0x00000000
reserved1 0
reserved2 0
Section
sectname __gopclntab
segname __DATA
addr 0x0000000000132920
size 0x000000000009a6de
offset 1255712
align 2^5 (32)
reloff 0
nreloc 0
flags 0x00000000
reserved1 0
reserved2 0
Section
sectname __noptrdata
segname __DATA
addr 0x00000000001cd000
size 0x000000000000cbdc
offset 1888256
align 2^5 (32)
reloff 0
nreloc 0
flags 0x00000000
reserved1 0
reserved2 0
Section
sectname __data
segname __DATA
addr 0x00000000001d9be0
size 0x0000000000006ad0
offset 1940448
align 2^5 (32)
reloff 0
nreloc 0
flags 0x00000000
reserved1 0
reserved2 0
Section
sectname __bss
segname __DATA
addr 0x00000000001e06c0
size 0x000000000001b830
offset 0
align 2^5 (32)
reloff 0
nreloc 0
flags 0x00000001
reserved1 0
reserved2 0
Section
sectname __noptrbss
segname __DATA
addr 0x00000000001fbf00
size 0x0000000000002518
offset 0
align 2^5 (32)
reloff 0
nreloc 0
flags 0x00000001
reserved1 0
reserved2 0
Load command 2
cmd LC_SEGMENT_64
cmdsize 72
segname __LINKEDIT
vmaddr 0x00000000001ff000
vmsize 0x0000000000072000
fileoff 1970176
filesize 465632
maxprot 0x00000007
initprot 0x00000001
nsects 0
flags 0x0
Load command 3
cmd LC_ID_DYLIB
cmdsize 112
name /var/folders/qv/lg43663s45jbmrkjvy64zh5r0000gn/T/go-build695961820/b001/exe/a.out.so (offset 24)
time stamp 1 Thu Jan 1 01:00:01 1970
current version 0.0.0
compatibility version 0.0.0
Load command 4
cmd LC_DYLD_INFO_ONLY
cmdsize 48
rebase_off 1970176
rebase_size 21536
bind_off 1991712
bind_size 65752
weak_bind_off 0
weak_bind_size 0
lazy_bind_off 2057464
lazy_bind_size 1080
export_off 2058544
export_size 43032
Load command 5
cmd LC_SYMTAB
cmdsize 24
symoff 2105928
nsyms 9259
stroff 2257224
strsize 178584
Load command 6
cmd LC_DYSYMTAB
cmdsize 80
ilocalsym 0
nlocalsym 6646
iextdefsym 6646
nextdefsym 2555
iundefsym 9201
nundefsym 58
tocoff 0
ntoc 0
modtaboff 0
nmodtab 0
extrefsymoff 0
nextrefsyms 0
indirectsymoff 2254072
nindirectsyms 788
extreloff 0
nextrel 0
locreloff 0
nlocrel 0
Load command 7
cmd LC_UUID
cmdsize 24
uuid 0CD073D4-4257-345E-8157-2ECBC871E244
Load command 8
cmd LC_VERSION_MIN_MACOSX
cmdsize 16
version 10.12
sdk 10.12
Load command 9
cmd LC_SOURCE_VERSION
cmdsize 16
version 0.0
Load command 10
cmd LC_LOAD_DYLIB
cmdsize 56
name /usr/lib/libSystem.B.dylib (offset 24)
time stamp 2 Thu Jan 1 01:00:02 1970
current version 1238.60.2
compatibility version 1.0.0
Load command 11
cmd LC_FUNCTION_STARTS
cmdsize 16
dataoff 2101576
datasize 4352
Load command 12
cmd LC_DATA_IN_CODE
cmdsize 16
dataoff 2105928
datasize 0
As you can see, no debug sections. Now that I think about it I knew this because I noticed when I changed the linker to make it able to emit debug symbols for plugins.
@aarzilli
First of all, I don't know how debugger works well. But It seems that it's impossible to debug golang plugin at this moment. Because as you said, the built plugin doesn't contain debugging symbols at all. Is this golang issue? Should I wait for new version of golang?
This is a major obstacle for me as I am working on same code base that uses plugins and I often switch between Linux and Mac while working on this.
Meantime I believe lack of debugging symbols in plugins on Darwin is still an issue in Go https://github.com/golang/go/issues/27502
@aarzilli : Thanks for your diligence in pushing this support forward. It's nice that https://github.com/golang/go/issues/27502 has now been fixed, but it seems like the ability to debug go plugins on Mac will still be blocked by https://github.com/golang/go/issues/25841, according to:
- your comment above, and
- your comment in https://github.com/golang/go/issues/27502.
Given this, perhaps you can revisit your comment in https://github.com/golang/go/issues/25841, where you responded that it was not important? Seems that @heschik was willing to dig into that issue before.
I discovered that the fix for https://github.com/golang/go/issues/27502 is present in go1.14beta1, so I installed that version:
go get golang.org/dl/go1.14beta1
go1.14beta1 download
Then I rebuilt the binaries from the example project:
go1.14beta1 build -gcflags 'all=-N -l' -buildmode=plugin -o eng/eng.so eng/greeter.go
go1.14beta1 build -gcflags 'all=-N -l' -o greeter greeter.go
I then used otool -l to validate that the eng.so binary had the DWARF debug sections.
✗ otool -l eng/eng.so | grep debug
sectname __zdebug_line
sectname __zdebug_pubname
sectname __zdebug_loc
sectname __zdebug_ranges
sectname __zdebug_pubtype
sectname __zdebug_aranges
sectname __zdebug_info
sectname __zdebug_frame
sectname __zdebug_abbrev
sectname __zdebug_str
So I got my hopes up, but when I tried debugging the greeter binary I hit the same problem.
✗ dlv --check-go-version=false exec ./greeter -- english
Type 'help' for list of commands.
(dlv) b greeter.go:58
Breakpoint 1 set at 0x40f33ac for main.main() ./greeter.go:58
(dlv) c
> main.main() ./greeter.go:58 (hits goroutine(1):1 total:1) (PC: 0x40f33ac)
53: fmt.Println("unexpected type from module symbol")
54: os.Exit(1)
55: }
56:
57: // 4. use the module
=> 58: greeter.Greet()
59:
60: }
(dlv) b eng/greeter.go:8
Command failed: Location "eng/greeter.go:8" not found
(dlv) s
Stopped at: 0x7ce92d0
=>no source available
(dlv) s
Hello Universe
Process has exited with status 0
(dlv)
I think it would behoove you to comment yourself on that thread, your first hand account of why you need it would be more persuasive than my second hand perception that some people might need it.
@aarzilli : I've put in a comment on that issue. But frankly I'm too ignorant about this topic to understand what the issues are here, and how exactly golang/go#25841 is blocking this issue.
e.g., I don't understand how you obtained the output that you cite, they look like they came from running readelf, but the ticket is about Mac. So I'm guessing maybe you targeted GOOS=linux in the build. And if so then I'm not seeing how that relates to this issue.
Furthermore, I seem to be seeing at least 2 different problems in my example above, but I'm not sure if they are symptoms of one underlying cause.
cannot set the breakpoint
On Mac, where it fails in this scenario, it looks like this:
(dlv) b eng/greeter.go:8
Command failed: Location "eng/greeter.go:8" not found
On linux where it works it looks like this:
(dlv) b eng/greeter.go:8
Breakpoint 2 set at 0x7f7d8c5c838f for plugin/unnamed-380bc3019cae9f99f0f646d134e7c21e1405b74f.greeting.Greet() ./eng/greeter.go:8
cannot step into the plugin code
On Mac. where it fails in this scenario, it looks like this:
(dlv) c
> main.main() ./greeter.go:58 (hits goroutine(1):1 total:1) (PC: 0x40f33ac)
53: fmt.Println("unexpected type from module symbol")
54: os.Exit(1)
55: }
56:
57: // 4. use the module
=> 58: greeter.Greet()
59:
60: }
(dlv) s
Stopped at: 0xcde92d0
=>no source available
On linux where it works it looks like this:
(dlv) c
> main.main() ./greeter.go:58 (hits goroutine(1):1 total:1) (PC: 0x533921)
53: fmt.Println("unexpected type from module symbol")
54: os.Exit(1)
55: }
56:
57: // 4. use the module
=> 58: greeter.Greet()
59:
60: }
(dlv) s
> plugin/unnamed-380bc3019cae9f99f0f646d134e7c21e1405b74f.(*greeting).Greet() <autogenerated>:1 (PC: 0x7f972c3cf421)
(dlv) s
> plugin/unnamed-380bc3019cae9f99f0f646d134e7c21e1405b74f.greeting.Greet() ./eng/greeter.go:7 (PC: 0x7f972c3cf381)
2:
3: import "fmt"
4:
5: type greeting string
6:
=> 7: func (g greeting) Greet() {
8: fmt.Println("Hello Universe")
9: }
10:
11: // exported
12: var Greeter greeting
(dlv) s
> plugin/unnamed-380bc3019cae9f99f0f646d134e7c21e1405b74f.greeting.Greet() ./eng/greeter.go:8 (PC: 0x7f972c3cf38f)
3: import "fmt"
4:
5: type greeting string
6:
7: func (g greeting) Greet() {
=> 8: fmt.Println("Hello Universe")
9: }
10:
11: // exported
12: var Greeter greeting
e.g., I don't understand how you obtained the output that you cite, they look like they came from running readelf
they are from objdump --dwarf, the GNU version, not the apple version.
Thanks for the response. Hmm, for some reason gobjdump (GNU objdump) doesn't show any details when run against a Mac-targeted binary. Maybe I'm missing some necessary flags for the gobjdump binary?
✗ go build -gcflags='-N -l' -buildmode pie -o greeter greeter.go
✗ /usr/local/Cellar/binutils/2.33.1/bin/gobjdump --dwarf greeter
greeter: file format mach-o-x86-64
Whereas when I crosscompile the binary for being run on linux (again from Mac), gobjdump --dwarf shows all kinds of fun dwarf details.
✗ GOOS=linux GOARCH=amd64 go build -gcflags '-N -l' -o greeter greeter.go
✗ /usr/local/Cellar/binutils/2.33.1/bin/gobjdump --dwarf greeter | head
greeter: file format elf64-x86-64
Contents of the .zdebug_abbrev section:
Number TAG (0x0)
1 DW_TAG_compile_unit [has children]
DW_AT_name DW_FORM_string
DW_AT_language DW_FORM_data1
DW_AT_stmt_list DW_FORM_sec_offset
...
NOTE: I wasn't able to run this linux-targeted build in
-buildmode pie, per the errors below:
✗ GOOS=linux GOARCH=amd64 go build -gcflags '-N -l' -buildmode pie -o greeter greeter.go
# command-line-arguments
loadinternal: cannot find runtime/cgo
/usr/local/Cellar/go/1.13.5/libexec/pkg/tool/darwin_amd64/link: running clang failed: exit status 1
ld: unknown option: -z
clang: error: linker command failed with exit code 1 (use -v to see invocation)
It's probably because of dwarf compression, you probably have to disable it with -ldflags='-compressdwarf=false'
@aarzilli : woot! That was it. Thanks for teaching me something.
✗ go build -gcflags='-N -l' -ldflags='-compressdwarf=false' -buildmode pie -o greeter greeter.go
✗ /usr/local/Cellar/binutils/2.33.1/bin/gobjdump --dwarf greeter
greeter: file format mach-o-x86-64
Raw dump of debug contents of section .debug_line:
Offset: 0x0
Length: 1288
DWARF Version: 2
...
After a long waiting, I walked around the problem by adding a static-linking mode to my plugin builder. With https://github.com/edwingeng/hotswap, you can switch freely between the dynamic-linking mode and the static-linking mode. Debugging a plugin under Mac is no longer a trouble to me.
P.S. Don't miss the --staticLinking argument.
@edwingeng debugging a go plugin should still be a issue for you. When i read trough your project you generate some wrapper code to compile code that was written as go plugin into the main binary ? Correct me if i am wrong but this is not what i would call a go plugin, for me its something compiled with buildmode=plugin that gets loaded at runtime. So still as long as this issue in https://github.com/golang/go/issues/25841 is not fixed by apple its hard to get this working. Alltough i saw commits from @aarzilli providing a workaround in delve https://github.com/go-delve/delve/pull/2374 but i am not sure if this fully gets debugging go plugins on macs working. Maybe @aarzilli can comment here on that :>
When i read trough your project you generate some wrapper code to compile code that was written as go plugin into the main binary ?
Yes.
for me its something compiled with buildmode=plugin that gets loaded at runtime.
If --staticLinking is NOT specified when building a plugin, it is still a perfect plugin, which is not debuggable on Mac. hotswap helps me debug the code in a plugin. It is a problem solver, not a curer:)
--staticLinking is a per build choice.
@FloThinksPi https://github.com/golang/go/issues/25841 seems to be fixed, should it be working now? I still get errors trying to debug a plugin in macos with delve.
Delve's side of this needs to be implemented. At the moment I don't have access to a working VM (and none of the builders have a version of macos that will receive the updated version of xcode) so I'm not going to do it any time soon, if someone else wants to do it: it should be fairly simple, just add a call to getLoadedDynamicLibraries at the end of ContinueOnce and call bi.AddImage for all the dynamic libraries.
I guess I'll give it a try since no one more experienced with delve and go is available.
@aarzilli Please give me a hand with this. I just tried your suggestion:
- Added the following code right before the return statement of
(*gbpProcess).ContinueOnce:
images, err := p.conn.getLoadedDynamicLibraries()
if err != nil {
return nil, stopReason, fmt.Errorf("could not load dynamic libraries %s", err)
}
for _, image := range images {
p.bi.AddImage(image.Pathname, image.LoadAddress)
}
go install- Still getting
could not find statement at /Users/rodrigo/testgo/plugin.go:8, please use a line with a statement
There must be something I'm missing.
Edit: I'm trying to use delve with dap.
Edit 2: Looks like AddImage is returning an error:
invalid magic number in record at byte 0x0
(The code I'm using to test this is just the example in the official docs https://pkg.go.dev/plugin)
@rstcruzo presumably it's coming from loadBinaryInfoMacho, you should check that it can actually load the plugin binary. Could be something in go's standard library.
I checked out the latest version of macOS (13.4) and it still has a bugged version of dsymutil.
@aarzilli I updated my macbook to macOS Sonoma. Is there a command I can run to check if dsymutil was updated?
No, build something and check that the debug_frame section is correct (you could probably find out from the version of llvm but I don't know which version has the fix).
FYI Sonoma (or rather the version of xcode that ships with sonoma) still has an old version of dsymutil.