parca-agent
parca-agent copied to clipboard
Published
20 hours ago •
parca-dev
parca-dev
libbpfgo: Libbpf v1.0 migration
Opening this issue to track the libbpf v1.0 migration:
- Variadic arguments bug https://github.com/parca-dev/parca-agent/issues/817
- Map issues:
caller=cpu.go:241 msg="failed to obtain profiles from eBPF maps" err="failed iterator: no such file or directory"
The update has only been partial so far, here's how to make a full one:
# Update Go module
go get github.com/aquasecurity/[email protected]
go mod tidy
# Update Git submodule
cd 3rdparty/libbpf
git fetch --tags
git checkout v1.0.0
# Build
cd ../..
make clean
make build
# Run
sudo dist/parca-agent --node=test
Everything works until you try to run it, the failed to obtain profiles from eBPF maps message is replaced by a SIGSEGV: segmentation violation:
Full logs
ooooooooo. .o. .
`888 `Y88. .888. .o8
888 .d88' .oooo. oooo d8b .ooooo. .oooo. .8"888. .oooooooo .ooooo. ooo. .oo. .o888oo
888ooo88P' `P )88b `888""8P d88' `"Y8 `P )88b .8' `888. 888' `88b d88' `88b `888P"Y88b 888
888 .oP"888 888 888 .oP"888 .88ooo8888. 888 888 888ooo888 888 888 888
888 d8( 888 888 888 .o8 d8( 888 .8' `888. `88bod8P' 888 .o 888 888 888 .
o888o `Y888""8o d888b `Y8bod8P' `Y888""8o o88o o8888o `8oooooo. `Y8bod8P' o888o o888o "888"
d" YD
"Y88888P'
level=info name=parca-agent ts=2022-09-21T19:52:56.116950609Z caller=main.go:203 msg="local profile storage is enabled" dir=./tmp/profiles
name=parca-agent ts=2022-09-21T19:52:56.116976112Z caller=main.go:210 msg=starting... node=test store=
fatal error: unexpected signal during runtime execution
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x186246d]
runtime stack:
runtime.throw({0x1d96060?, 0x7fd0bc0008fa?})
runtime/panic.go:992 +0x71
runtime.sigpanic()
runtime/signal_unix.go:802 +0x389
goroutine 13 [syscall]:
runtime.cgocall(0x17c5be0, 0xc0000af858)
runtime/cgocall.go:157 +0x5c fp=0xc0000af830 sp=0xc0000af7f8 pc=0x40525c
github.com/aquasecurity/libbpfgo._C2func_bpf_object__open_mem(0x7fd0bc000b80, 0x700, 0xc000306280)
_cgo_gotypes.go:325 +0x57 fp=0xc0000af858 sp=0xc0000af830 pc=0x17a83b7
github.com/aquasecurity/libbpfgo.NewModuleFromBufferArgs.func3(0x0?, 0x0?, 0x1d6f465?)
github.com/aquasecurity/[email protected]/libbpfgo.go:472 +0x7e fp=0xc0000af8b0 sp=0xc0000af858 pc=0x17aca1e
github.com/aquasecurity/libbpfgo.NewModuleFromBufferArgs({{0x0, 0x0}, {0x1d6f465, 0x17}, {0x1d51a75, 0x5}, {0x0, 0x0}, {0x2dc0660, 0x700, ...}})
github.com/aquasecurity/[email protected]/libbpfgo.go:472 +0x1f6 fp=0xc0000af9a8 sp=0xc0000af8b0 pc=0x17ac636
github.com/parca-dev/parca-agent/pkg/profiler/cpu.(*CPU).Run(0xc000326e00, {0x2007be8?, 0xc00007a9c0})
github.com/parca-dev/parca-agent/pkg/profiler/cpu/cpu.go:145 +0x1d8 fp=0xc0000afe28 sp=0xc0000af9a8 pc=0x17b9d58
main.run.func6()
github.com/parca-dev/parca-agent/cmd/parca-agent/main.go:315 +0x388 fp=0xc0000aff88 sp=0xc0000afe28 pc=0x17c25e8
github.com/oklog/run.(*Group).Run.func1({0xc00007aa00?, 0xc0004dba10?})
github.com/oklog/[email protected]/group.go:38 +0x2f fp=0xc0000affc0 sp=0xc0000aff88 pc=0x99412f
github.com/oklog/run.(*Group).Run.func2()
github.com/oklog/[email protected]/group.go:39 +0x2e fp=0xc0000affe0 sp=0xc0000affc0 pc=0x9940ce
runtime.goexit()
runtime/asm_amd64.s:1571 +0x1 fp=0xc0000affe8 sp=0xc0000affe0 pc=0x46cc81
created by github.com/oklog/run.(*Group).Run
github.com/oklog/[email protected]/group.go:37 +0x22a
goroutine 1 [chan receive]:
github.com/oklog/run.(*Group).Run(0xc000a1fab8)
github.com/oklog/[email protected]/group.go:43 +0x7c
main.run({0x1febda0?, 0xc0003065f0}, 0xc000306640, {{0xc00088b0c0, 0x4}, {0xc00088b0d0, 0x5}, {0xc00088b0b0, 0x4}, 0x2540be400, ...})
github.com/parca-dev/parca-agent/cmd/parca-agent/main.go:338 +0x2827
main.main()
github.com/parca-dev/parca-agent/cmd/parca-agent/main.go:120 +0x313
goroutine 26 [select]:
github.com/baidubce/bce-sdk-go/util/log.NewLogger.func1()
github.com/baidubce/[email protected]/util/log/logger.go:375 +0xb9
created by github.com/baidubce/bce-sdk-go/util/log.NewLogger
github.com/baidubce/[email protected]/util/log/logger.go:368 +0x136
goroutine 7 [select]:
go.opencensus.io/stats/view.(*worker).start(0xc0001a6c00)
[email protected]/stats/view/worker.go:276 +0xad
created by go.opencensus.io/stats/view.init.0
[email protected]/stats/view/worker.go:34 +0x8d
goroutine 33 [chan receive]:
github.com/goburrow/cache.(*localCache).processEntries(0xc00023e000)
github.com/goburrow/[email protected]/local.go:212 +0x8b
created by github.com/goburrow/cache.(*localCache).init
github.com/goburrow/[email protected]/local.go:83 +0x25b
goroutine 87 [chan receive]:
github.com/goburrow/cache.(*localCache).processEntries(0xc00023e2c0)
github.com/goburrow/[email protected]/local.go:212 +0x8b
created by github.com/goburrow/cache.(*localCache).init
github.com/goburrow/[email protected]/local.go:83 +0x25b
goroutine 88 [select]:
github.com/parca-dev/parca-agent/pkg/discovery.(*SystemdDiscoverer).Run(0xc0004dafb0, {0x2007be8, 0xc00007a740}, 0xc000126088?)
github.com/parca-dev/parca-agent/pkg/discovery/systemd.go:56 +0x145
github.com/parca-dev/parca-agent/pkg/discovery.(*Manager).startProvider.func1()
github.com/parca-dev/parca-agent/pkg/discovery/discovery_manager.go:220 +0x52
created by github.com/parca-dev/parca-agent/pkg/discovery.(*Manager).startProvider
github.com/parca-dev/parca-agent/pkg/discovery/discovery_manager.go:219 +0x38e
goroutine 89 [select]:
github.com/parca-dev/parca-agent/pkg/discovery.(*Manager).updater(0xc000128510, {0x2007be8, 0xc00007a740}, 0xc0000ce6e0, 0xc0002a0a20)
github.com/parca-dev/parca-agent/pkg/discovery/discovery_manager.go:229 +0xd9
created by github.com/parca-dev/parca-agent/pkg/discovery.(*Manager).startProvider
github.com/parca-dev/parca-agent/pkg/discovery/discovery_manager.go:224 +0x430
goroutine 90 [chan receive]:
github.com/goburrow/cache.(*localCache).processEntries(0xc00023e580)
github.com/goburrow/[email protected]/local.go:212 +0x8b
created by github.com/goburrow/cache.(*localCache).init
github.com/goburrow/[email protected]/local.go:83 +0x25b
goroutine 92 [chan receive]:
github.com/goburrow/cache.(*localCache).processEntries(0xc00023e6e0)
github.com/goburrow/[email protected]/local.go:212 +0x8b
created by github.com/goburrow/cache.(*localCache).init
github.com/goburrow/[email protected]/local.go:83 +0x25b
goroutine 95 [chan receive]:
github.com/goburrow/cache.(*localCache).processEntries(0xc00023e840)
github.com/goburrow/[email protected]/local.go:212 +0x8b
created by github.com/goburrow/cache.(*localCache).init
github.com/goburrow/[email protected]/local.go:83 +0x25b
goroutine 96 [chan receive]:
github.com/goburrow/cache.(*localCache).processEntries(0xc00023e9a0)
github.com/goburrow/[email protected]/local.go:212 +0x8b
created by github.com/goburrow/cache.(*localCache).init
github.com/goburrow/[email protected]/local.go:83 +0x25b
goroutine 97 [chan receive]:
github.com/goburrow/cache.(*localCache).processEntries(0xc00023eb00)
github.com/goburrow/[email protected]/local.go:212 +0x8b
created by github.com/goburrow/cache.(*localCache).init
github.com/goburrow/[email protected]/local.go:83 +0x25b
goroutine 98 [chan receive]:
github.com/goburrow/cache.(*localCache).processEntries(0xc00023ec60)
github.com/goburrow/[email protected]/local.go:212 +0x8b
created by github.com/goburrow/cache.(*localCache).init
github.com/goburrow/[email protected]/local.go:83 +0x25b
goroutine 114 [chan receive]:
github.com/parca-dev/parca-agent/pkg/discovery.(*Manager).Run(0xc000128510, {0x2007be8?, 0xc00007a500})
github.com/parca-dev/parca-agent/pkg/discovery/discovery_manager.go:147 +0xaf
main.run.func3()
github.com/parca-dev/parca-agent/cmd/parca-agent/main.go:244 +0x228
github.com/oklog/run.(*Group).Run.func1({0xc0006dc270?, 0xc0004db200?})
github.com/oklog/[email protected]/group.go:38 +0x2f
created by github.com/oklog/run.(*Group).Run
github.com/oklog/[email protected]/group.go:37 +0x22a
goroutine 115 [select]:
github.com/parca-dev/parca-agent/pkg/discovery.(*Manager).sender(0xc000128510, {0x2007be8, 0xc00007a500})
github.com/parca-dev/parca-agent/pkg/discovery/discovery_manager.go:256 +0xe5
created by github.com/parca-dev/parca-agent/pkg/discovery.(*Manager).Run
github.com/parca-dev/parca-agent/pkg/discovery/discovery_manager.go:146 +0x95
goroutine 14 [IO wait]:
internal/poll.runtime_pollWait(0x7fd0d2e321d8, 0x72)
runtime/netpoll.go:302 +0x89
internal/poll.(*pollDesc).wait(0xc000328a80?, 0x0?, 0x0)
internal/poll/fd_poll_runtime.go:83 +0x32
internal/poll.(*pollDesc).waitRead(...)
internal/poll/fd_poll_runtime.go:88
internal/poll.(*FD).Accept(0xc000328a80)
internal/poll/fd_unix.go:614 +0x22c
net.(*netFD).accept(0xc000328a80)
net/fd_unix.go:172 +0x35
net.(*TCPListener).accept(0xc000674000)
net/tcpsock_posix.go:139 +0x28
net.(*TCPListener).Accept(0xc000674000)
net/tcpsock.go:288 +0x3d
net/http.(*Server).Serve(0xc000390540, {0x2006d38, 0xc000674000})
net/http/server.go:3039 +0x385
net/http.Serve(...)
net/http/server.go:2543
main.run.func8()
github.com/parca-dev/parca-agent/cmd/parca-agent/main.go:331 +0x265
github.com/oklog/run.(*Group).Run.func1({0xc00078c030?, 0xc000674018?})
github.com/oklog/[email protected]/group.go:38 +0x2f
created by github.com/oklog/run.(*Group).Run
github.com/oklog/[email protected]/group.go:37 +0x22a
goroutine 15 [select]:
github.com/oklog/run.SignalHandler.func1()
github.com/oklog/[email protected]/actors.go:18 +0xe9
github.com/oklog/run.(*Group).Run.func1({0xc00078c060?, 0xc00053a080?})
github.com/oklog/[email protected]/group.go:38 +0x2f
created by github.com/oklog/run.(*Group).Run
github.com/oklog/[email protected]/group.go:37 +0x22a
goroutine 130 [syscall]:
os/signal.signal_recv()
runtime/sigqueue.go:151 +0x2f
os/signal.loop()
os/signal/signal_unix.go:23 +0x19
created by os/signal.Notify.func1.1
os/signal/signal.go:151 +0x2a
If you update libbpfgo to include the fix from aquasecurity/libbpfgo#248:
go get github.com/aquasecurity/[email protected]
go mod tidy
It seems to be a lot better, but what use to be a deprecation warning before is now an error in v1:
libbpf: elf: legacy map definitions in 'maps' section are not supported by libbpf v1.0+
Failed to open bpf object: Operation not supported
level=error name=parca-agent ts=2022-09-21T20:03:41.041680823Z caller=main.go:121 err="new bpf module: failed to open BPF object parca: operation not supported"
Looks like aya still does not support the new format, latest update I could find in https://github.com/aya-rs/aya/pull/248#discussion_r860845909:
Support in
aya-bpffor BTF maps is sadly not as simple as the BTF generated for Rust programs by LLVM isn't correct.
