parca-agent icon indicating copy to clipboard operation
parca-agent copied to clipboard

arm64 dwarf unwind wrong

Open zdyj3170101136 opened this issue 1 year ago • 7 comments

go program, use fp to unwind in arm64 have bug:

https://github.com/golang/go/issues/70804.

use parca-agent v0.31.1 to unwind.

program:

package main

import (
        "encoding/json"
        "net/http"
        _ "net/http/pprof"
)

func main() {
        go func() {
                http.ListenAndServe(":7079", nil )
        }()
        for {
                json.Marshal([]byte("sss"))
        }
}

cmd:

./parca-agent_0.31.1_Linux_aarch64 --local-store-directory=pprof --profiling-cpu-sampling-frequency=100 --profiling-duration=60s --dwarf-unwinding-mixed

the parca-agent output:

go tool pprof -svg
截屏2024-12-20 下午4 27 09

zdyj3170101136 avatar Dec 20 '24 08:12 zdyj3170101136

Could you try with latest Parca agent? We actually changed the unwind strategy dramatically for go starting in v0.32.

brancz avatar Dec 20 '24 09:12 brancz

Could you try with latest Parca agent? We actually changed the unwind strategy dramatically for go starting in v0.32.

截屏2024-12-20 下午5 34 01

same as before.

zdyj3170101136 avatar Dec 20 '24 09:12 zdyj3170101136

cat /proc/version
Linux version 5.10.134-17.3.al8.aarch64 ([email protected]) (gcc (GCC) 10.2.1 20200825 (Alibaba 10.2.1-3.8 2.32), GNU ld version2.35-12.3.al8) #1 SMP Thu Oct 31 14:27:09 CST 2024

the perf report could get correct result.

perf record -a -g --call-graph=dwarf -F 100
perf report
截屏2024-12-20 下午5 37 39

zdyj3170101136 avatar Dec 20 '24 09:12 zdyj3170101136

@brancz

the reason is that the go exec do not have eh_frame section.

if we could parse debug_frame , we could unwind stack by using ld register.

i already make a smallest demo and it run success.

I am happy to contribute, but the opentelemetry ebpf agent is too complicated that i could not read it.

zdyj3170101136 avatar Dec 30 '24 11:12 zdyj3170101136

just noticed it has already implement debug_frame in v0.31.0.

but it work on v0.30.0, strange.

zdyj3170101136 avatar Dec 30 '24 14:12 zdyj3170101136

Have you tried the v0.32+ agent? We've observed arm64 working flawlessly.

brancz avatar Jan 06 '25 16:01 brancz

Have you tried the v0.32+ agent? We've observed arm64 working flawlessly.

finally i find the reason.

func bpf_get_stack seems work not right.

in this mr, parca agent implement it's fp unwind, it work:

https://github.com/parca-dev/parca-agent/pull/2419/files

zdyj3170101136 avatar Jan 07 '25 02:01 zdyj3170101136