parca-agent icon indicating copy to clipboard operation
parca-agent copied to clipboard

GraalVM DWARF unwinding not working

Open brancz opened this issue 2 years ago • 1 comments

Describe the bug

The DWARF unwinder doesn't appear to be working with GraalVM binaries. The framepointer unwinder works so not a big deal at the moment, but we should still understand why and fix it.

To Reproduce

Take the Quarkus example: https://github.com/parca-dev/parca-demo/tree/main/graalvm-quarkus

And remove the -H:+PreserveFramePointer flag from https://github.com/parca-dev/parca-demo/blob/main/graalvm-quarkus/src/main/resources/application.properties

And build with podman build . in the root of the Quarkus example directory, then run it with podman run --rm -it -p8080:8080 <image> on the same host as Parca Agent is running and run a loop of curl http://localhost:8080/hello against it to produce some CPU usage.

Expected behavior

Enabling frame-pointers wouldn't be necessary and the DWARF unwinder would just work.

Screenshots or Profiles (using https://pprof.me)

https://pprof.me/6e9df09

Screenshot 2023-08-08 at 18 16 24

Logs

Verbose BPF logs: https://gist.github.com/brancz/9a6ae5c2eeb78cba1cd661196e603fae

Nothing noticeable in the agent userspace logs, supposedly the unwind tables are even compiled successfully:

level=debug name=parca-agent ts=2023-08-08T15:26:23.308026831Z caller=maps.go:975 component=cpu_profiler component=bpf_maps msg="dealing with main object" mapping="ExecutableMapping {LoadAddr: 0x400000, StartAddr: 0x400000, EndAddr: 0x2da2000, Executable:/usr/bin/quarkus-graal}"
level=debug name=parca-agent ts=2023-08-08T15:26:23.308045085Z caller=maps.go:984 component=cpu_profiler component=bpf_maps msg="adding memory mappings in for executable" executableID=49 buildID=4cc415822512c8cd1f4fdb6a82be16e5170c6f01 executable=/usr/bin/quarkus-graal
level=debug name=parca-agent ts=2023-08-08T15:26:23.308061716Z caller=maps.go:742 component=cpu_profiler component=bpf_maps msg="mapping caching, seen before" buildID=4cc415822512c8cd1f4fdb6a82be16e5170c6f01
level=debug name=parca-agent ts=2023-08-08T15:26:23.308073599Z caller=maps.go:923 component=cpu_profiler component=bpf_maps msg="setUnwindTable called" shards=1 maxshards=50 sumofunwindrows=282542

Software (please complete the following information):

  • Parca Agent Version: v0.23.3
  • Parca Server Version (if applicable): Polar Signals Cloud

Workload (please complete the following information):

  • Runtime (if applicable): quay.io/quarkus/ubi-quarkus-graalvmce-builder-image:22.3-java17
  • Compiler (if applicable): Graal

Environment (please complete the following information):

  • Linux Distribution: [cat /etc/*-release] NixOS
  • Linux Version: [uname -r] 6.3
  • Arch: [uname -m] amd64
  • Kubernetes Version (if applicable): [kubectl --version] n/a
  • Container Runtime (if applicable): n/a

brancz avatar Aug 08 '23 16:08 brancz

As mentioned in #617, it turns out this is likely due to Graal only producing .debug_frame, which we don't have support for (yet).

brancz avatar Aug 09 '23 14:08 brancz