Test failed on arm64: TestCgoStacktrace and TestCgoEval

[zhsj]

1. What version of Delve are you using (dlv version)?

$ git log --oneline -1
9a3c9eba (HEAD -> master, origin/master, origin/HEAD) proc/*: add launch option to disable ASLR (#2202)

2. What version of Go are you using? (go version)?

$ go version
go version go1.15.2 linux/arm64

3. What operating system and processor architecture are you using?

$ cat /etc/os-release 
PRETTY_NAME="Debian GNU/Linux bullseye/sid"
NAME="Debian GNU/Linux"
ID=debian
HOME_URL="https://www.debian.org/"
SUPPORT_URL="https://www.debian.org/support"
BUG_REPORT_URL="https://bugs.debian.org/"

$ lscpu
Architecture:        aarch64
Byte Order:          Little Endian
CPU(s):              4
On-line CPU(s) list: 0-3
Thread(s) per core:  1
Core(s) per socket:  4
Socket(s):           1
NUMA node(s):        1
Vendor ID:           APM
Model:               1
Model name:          X-Gene
Stepping:            0x0
BogoMIPS:            100.00
NUMA node0 CPU(s):   0-3
Flags:               fp asimd evtstrm cpuid

4. What did you do?

I'm going to package delve for Debian. But the tests fail on arm64. https://buildd.debian.org/status/fetch.php?pkg=delve&arch=arm64&ver=1.5.0-4&stamp=1603213253&raw=0

It's also reproducible with master.

5. What did you expect to see?

The tests succeed.

6. What did you see instead?

(sid_arm64-dchroot)zhsj@amdahl:~/t/delve$ go test -mod=vendor -v ./service/test/ -run TestCgoEval                                                                          
=== RUN   TestCgoEval
    support.go:248: enabling recording for github.com/go-delve/delve/service/test_test.TestCgoEval                                                                         
    variables_test.go:68: Expected "\"a string\"" got "(unreadable could not read string at 0x8020080280200802 due to input/output error)" (for variable s)                
--- FAIL: TestCgoEval (1.71s)
FAIL
FAIL    github.com/go-delve/delve/service/test  1.738s
FAIL
(sid_arm64-dchroot)zhsj@amdahl:~/t/delve$ go test -mod=vendor -v ./pkg/proc -run TestCgoStacktrace                                                                          
=== RUN   TestCgoStacktrace
    proc_test.go:3347: iteration step 0
    proc_test.go:3211:  0x48ddf0 -0x90 -0xb8 main.main at main.go:13
    proc_test.go:3211:  0x430d7c -0x30 -0x13898 runtime.main at proc.go:204
    proc_test.go:3211:  0x45c574 -0x30 -0x4000044000 runtime.goexit at asm_arm64.s:1136
    proc_test.go:3347: iteration step 1
    proc_test.go:3211:  0x48e178 0xffffcd2f5660 0xffffcd2f5650 C.helloworld_pt2 at hello.c:15
    proc_test.go:3211:  0x48e1a8 0xffffcd2f5680 0xffffcd2f5670 C.helloworld at hello.c:19
    proc_test.go:3211:  0x48e130 0xffffcd2f56a0 0xffffcd2f5690 C._cgo_03f30c882f33_Cfunc_helloworld at cgo-gcc-prolog:50
    proc_test.go:3211:  0x45c33c -0x4000043f48 -0x4000044000 runtime.asmcgocall at asm_arm64.s:922
    proc_test.go:3211:  0x0 0x0 0x0 ? at .:0
    proc_test.go:3266: couldn't find frame 2 [C.helloworld_pt2 C.helloworld main.main]
    proc_test.go:3267: expected: [C.helloworld_pt2 C.helloworld main.main]
    proc_test.go:3392: see previous loglines
--- FAIL: TestCgoStacktrace (2.01s)
=== RUN   TestCgoStacktrace2
    proc_test.go:75: skipped on arm64: broken
--- SKIP: TestCgoStacktrace2 (0.00s)
FAIL
FAIL    github.com/go-delve/delve/pkg/proc      2.050s
FAIL

[oxisto]

BTW, these also fail on darwin/arm64, so it is not linux specific. I can take a look at this over the weekend, now the lldb darwin/arm64 stuff is merged in.

[aarzilli]

Yes, I was thinking about this the other day. The way we are retrieving the location of the current goroutine (GAddr) on arm64 is just wrong. It's in the general purpose register while Go code is executed but C code can overwrite it and the runtime saves it to TPIDR_EL0 when doing cgo calls. That's where we should be loading it from. I don't know if it's possible with the gdbserial backend.

[oxisto]

Yes, I was thinking about this the other day. The way we are retrieving the location of the current goroutine (GAddr) on arm64 is just wrong. It's in the general purpose register while Go code is executed but C code can overwrite it and the runtime saves it to TPIDR_EL0 when doing cgo calls. That's where we should be loading it from. I don't know if it's possible with the gdbserial backend.

Hmm probably not, at least not at first glance. For macOS it seems to be stored in the TPIDRRO_EL0 register (or at least it can be accessed using that register), and the only way to retrieve it would be using a MRS command to load it from the system register into a general purpose register. That is also how Go does it:

https://github.com/golang/go/blob/846dce9d05f19a1f53465e62a304dea21b99f910/src/runtime/tls_arm64.h#L26

So in theory one could do a similar hack as to load the GAddr on amd64 by allocating some memory with the MRS instruction, jumping to it and cleaning up afterwards. Probably very hacky and I am not quite sure which register to save the value to because they might all be used by the C stack. Meh.

Update: Ah, well I could just, i.e. use X0, save the existing value of that register in Delve's memory, jump to the MRS instruction, restore X0.

[oxisto]

@aarzilli Is there any "easy" way to check, whether we are in a cgo stack, i.e. some convenience function somewhere in Delve?

[aarzilli]

Not without being able to read the current goroutine, it's impossible.

[oxisto]

Not without being able to read the current goroutine, it's impossible.

That sounds like a chicken-and-egg problem then :)

[oxisto]

After a little bit of investigation it seems that it is not a problem of goroutines, but rather how the stack is being calculated on arm in Delve. The code is a little bit inconsistent with the one for AMD64, I managed to get reconstruct more frames in those test cases but still at some point the return address is 0 although the stack is not at the "end" yet. I will investigate this a little more but I think it will take some time.

[aarzilli]

That code is pretty complicated, if you need help ask.

in theory one could do a similar hack as to load the GAddr on amd64 by allocating some memory with the MRS instruction, jumping to it and cleaning up afterwards. Probably very hacky and I am not quite sure which register to save the value to because they might all be used by the C stack

the amd64 code has the same issue, it doesn't matter which one you choose because you would restore it after reading.