parca-dev
Unusable debug info is uploaded
I have both parca and parca-agent running on the same machine.
I also have dockerd running on the same machine and I get this flamegraph:
I have debug info available for dockerd:
ivan@cube:~$ file /usr/bin/dockerd
/usr/bin/dockerd: ELF 64-bit LSB pie executable, x86-64, version 1 (SYSV), dynamically linked, interpreter /lib64/ld-linux-x86-64.so.2, BuildID[sha1]=a52a3c0947657f986c65095f7bbb4e29ab53343f, for GNU/Linux 3.2.0, with debug_info, not stripped
I can also see debug info uploaded:
ivan@cube:~$ cat ~/projects/parca/data/debuginfo/a52a3c0947657f986c65095f7bbb4e29ab53343f/metadata
{
"buildId": "a52a3c0947657f986c65095f7bbb4e29ab53343f",
"source": "SOURCE_UPLOAD",
"upload": {
"id": "a4518131-1c07-4e53-a188-092bf99c80fc",
"hash": "41a765552e25adb26da340ed5e39d3e7",
"state": "STATE_UPLOADED",
"startedAt": "2025-01-27T06:01:30.695240535Z",
"finishedAt": "2025-01-27T06:01:30.840336583Z"
},
"quality": {
"hasDwarf": true,
"hasSymtab": true,
"hasDynsym": true
}
ivan@cube:~$ file~/projects/parca/data/debuginfo/a52a3c0947657f986c65095f7bbb4e29ab53343f/debuginfo
/home/ivan/projects/parca/data/debuginfo/a52a3c0947657f986c65095f7bbb4e29ab53343f/debuginfo: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), statically linked, BuildID[sha1]=a52a3c0947657f986c65095f7bbb4e29ab53343f, for GNU/Linux 3.2.0, with debug_info, not stripped
Let's pick one addr (0x22cda8d) and consult debug info with addr2line just to be sure:
ivan@cube:~$ eu-addr2line -f -e ~/projects/parca/data/debuginfo/a52a3c0947657f986c65095f7bbb4e29ab53343f/debuginfo 0x22cda8d
??
??:0
No luck! Not even with go tool addr2line:
ivan@cube:~$ eu-addr2line -f -e ~/projects/parca/data/debuginfo/a52a3c0947657f986c65095f7bbb4e29ab53343f/debuginfo 0x22cda8d
??
??:0
Now here's the fun thing, running the same thing on the original binary works:
ivan@cube:~$ eu-addr2line -f -e /usr/bin/dockerd 0x22cda8d
github.com/docker/docker/api/server.(*Server).makeHTTPHandler.func1
/_/GOROOT/src/runtime/cgo/gcc_mmap.c:37:3
ivan@cube:~$ echo 0x22cda8d | go tool addr2line /usr/bin/dockerd
github.com/docker/docker/api/server.(*Server).makeHTTPHandler.func1
/root/build-deb/engine/api/server/server.go:56
It turns out that the uploaded file is quite a bit smaller than the original:
ivan@cube:~$ ls -l ~/projects/parca/data/debuginfo/a52a3c0947657f986c65095f7bbb4e29ab53343f/debuginfo /usr/bin/dockerd
-rw-rw-r-- 1 ivan ivan 38452051 Jan 27 06:01 /home/ivan/projects/parca/data/debuginfo/a52a3c0947657f986c65095f7bbb4e29ab53343f/debuginfo
-rwxr-xr-x 2 root root 107762880 Sep 20 11:41 /usr/bin/dockerd
Here's the diff in section sizes:
ivan@cube:~$ diff -rup <(size -A /usr/bin/dockerd) <(size -A /home/ivan/projects/parca/data/debuginfo/a52a3c0947657f986c65095f7bbb4e29ab53343f/debuginfo)
--- /dev/fd/63 2025-01-27 06:09:05.454218257 +0000
+++ /dev/fd/62 2025-01-27 06:09:05.454218257 +0000
@@ -1,36 +1,36 @@
-/usr/bin/dockerd :
+/home/ivan/projects/parca/data/debuginfo/a52a3c0947657f986c65095f7bbb4e29ab53343f/debuginfo :
section size addr
-.interp 28 848
+.interp 0 848
.note.gnu.property 32 880
.note.gnu.build-id 36 912
.note.ABI-tag 32 948
.note.go.buildid 100 980
-.gnu.hash 1254760 1080
+.gnu.hash 0 1080
.dynsym 5171328 1255840
.dynstr 8442653 6427168
-.gnu.version 430944 14869822
-.gnu.version_r 144 15300768
+.gnu.version 0 14869822
+.gnu.version_r 0 15300768
.rela.dyn 7906632 15300912
.rela.plt 2088 23207544
-.init 23 23212032
+.init 0 23212032
.plt 1408 23212064
.plt.got 8 23213472
-.text 31684652 23213504
-.fini 9 54898156
-.rodata 4712716 54898688
-.eh_frame_hdr 724 59611404
-.eh_frame 3636 59612128
-.tbss 8 59619248
-.init_array 8 59619248
-.fini_array 8 59619256
-.data.rel.ro 29827888 59619264
-.dynamic 512 89447152
-.got 776 89447664
-.data 800904 89448448
+.text 0 23213504
+.fini 0 54898156
+.rodata 0 54898688
+.eh_frame_hdr 0 59611404
+.eh_frame 0 59612128
+.tbss 0 59619248
+.init_array 0 59619248
+.fini_array 0 59619256
+.data.rel.ro 0 59619264
+.dynamic 0 89447152
+.got 0 89447664
+.data 0 89448448
.go.buildinfo 10832 90249360
-.noptrdata 585943 90260192
-.bss 470784 90846144
-.noptrbss 85584 91316928
+.noptrdata 0 90260192
+.bss 0 90846144
+.noptrbss 0 91316928
.comment 31 0
.debug_aranges 387 0
.debug_info 15587 0
@@ -40,6 +40,6 @@ section size addr
.debug_line_str 688 0
.debug_loclists 3434 0
.debug_rnglists 212 0
-Total 91423053
+Total 21563002
I'm not quite sure which section is missing, but something important is definitely being skipped.
This is on latest main (68dcb27ca4292ca7e7ff38a80ec22d1b0fa3fecb).
Thank you for reporting!
The agent attempts to extract just the debuginfo, not the actual executable section. It seems like there are others that we're missing that we shouldn't be zeroing or throwing away.
Could you take the original binary, and perform objdump --only-keep-debug on it, and perform the same diff. That should give us an initial clue as to which sections we differ from other tooling when we extract, then adding those sections not to be skipped/zeroed should be trivial.
If you can't wait to debug this, you can ad the --debuginfo-strip=false flag to your agent and it will upload the original binary as is.
Copying just debug:
ivan@cube:~$ objcopy --only-keep-debug /usr/bin/dockerd /tmp/dockerd.debug
Produces a smaller file than parca does:
ivan@cube:~$ ls -l ~/projects/parca/data/debuginfo/a52a3c0947657f986c65095f7bbb4e29ab53343f/debuginfo /usr/bin/dockerd /tmp/dockerd.debug
-rw-rw-r-- 1 ivan ivan 38452051 Jan 27 06:01 /home/ivan/projects/parca/data/debuginfo/a52a3c0947657f986c65095f7bbb4e29ab53343f/debuginfo
-rwxrwxr-x 1 ivan ivan 16917824 Jan 30 04:29 /tmp/dockerd.debug
-rwxr-xr-x 2 root root 107762880 Sep 20 11:41 /usr/bin/dockerd
It's still usable for addr2line:
ivan@cube:~$ eu-addr2line -f -e /tmp/dockerd.debug 0x22cda8d
github.com/docker/docker/api/server.(*Server).makeHTTPHandler.func1
/_/GOROOT/src/runtime/cgo/gcc_mmap.c:37:3
Here's the diff between parca and objcopy:
ivan@cube:~$ diff -rup <(size -A ~/projects/parca/data/debuginfo/a52a3c0947657f986c65095f7bbb4e29ab53343f/debuginfo) <(size -A /tmp/dockerd.debug)
--- /dev/fd/63 2025-01-30 04:31:31.450741826 +0000
+++ /dev/fd/62 2025-01-30 04:31:31.449741830 +0000
@@ -1,36 +1,36 @@
-/home/ivan/projects/parca/data/debuginfo/a52a3c0947657f986c65095f7bbb4e29ab53343f/debuginfo :
+/tmp/dockerd.debug :
section size addr
-.interp 0 848
+.interp 28 848
.note.gnu.property 32 880
.note.gnu.build-id 36 912
.note.ABI-tag 32 948
.note.go.buildid 100 980
-.gnu.hash 0 1080
+.gnu.hash 1254760 1080
.dynsym 5171328 1255840
.dynstr 8442653 6427168
-.gnu.version 0 14869822
-.gnu.version_r 0 15300768
+.gnu.version 430944 14869822
+.gnu.version_r 144 15300768
.rela.dyn 7906632 15300912
.rela.plt 2088 23207544
-.init 0 23212032
+.init 23 23212032
.plt 1408 23212064
.plt.got 8 23213472
-.text 0 23213504
-.fini 0 54898156
-.rodata 0 54898688
-.eh_frame_hdr 0 59611404
-.eh_frame 0 59612128
-.tbss 0 59619248
-.init_array 0 59619248
-.fini_array 0 59619256
-.data.rel.ro 0 59619264
-.dynamic 0 89447152
-.got 0 89447664
-.data 0 89448448
+.text 31684652 23213504
+.fini 9 54898156
+.rodata 4712716 54898688
+.eh_frame_hdr 724 59611404
+.eh_frame 3636 59612128
+.tbss 8 59619248
+.init_array 8 59619248
+.fini_array 8 59619256
+.data.rel.ro 29827888 59619264
+.dynamic 512 89447152
+.got 776 89447664
+.data 800904 89448448
.go.buildinfo 10832 90249360
-.noptrdata 0 90260192
-.bss 0 90846144
-.noptrbss 0 91316928
+.noptrdata 585943 90260192
+.bss 470784 90846144
+.noptrbss 85584 91316928
.comment 31 0
.debug_aranges 387 0
.debug_info 15587 0
@@ -40,6 +40,6 @@ section size addr
.debug_line_str 688 0
.debug_loclists 3434 0
.debug_rnglists 212 0
-Total 21563002
+Total 91423053
And here's the diff between the original binary and objdump, which you might find interesting:
ivan@cube:~$ diff -rup <(size -A /usr/bin/dockerd) <(size -A /tmp/dockerd.debug)
--- /dev/fd/63 2025-01-30 04:32:53.087395343 +0000
+++ /dev/fd/62 2025-01-30 04:32:53.088395339 +0000
@@ -1,4 +1,4 @@
-/usr/bin/dockerd :
+/tmp/dockerd.debug :
section size addr
.interp 28 848
.note.gnu.property 32 880
Here's the docker package URL if you want to play around with it locally:
- https://download.docker.com/linux/debian/dists/bookworm/pool/stable/amd64/docker-ce_27.3.1-1%7edebian.12%7ebookworm_amd64.deb
Non-stripped upload needed some fixing: #3031.
Even with full debug info uploaded, parca is unable to symbolize the stacks. Luckily with https://github.com/parca-dev/parca/pull/5299 that can be fixed: