Use `-ffile-prefix-map` to get consistent debug info even across different users and machines

[jyn514]

This is a more principled fix to #163 than #168. It avoids non-determinism from the tempdir where the header files are stored, as well as differences from user home directories and project directories.

Ideally, this would just omit the debug info altogether, but libbpf-sys gives a hard error if it's missing. I tried stripping the object files with -Wl,-s, but it didn't remove the non-deterministic data.

Fixes #163. Closes #168.

Signed-off-by: Joshua Nelson [email protected]

[jyn514]

I tested this with readelf -x .debug_line:

Hex dump of section '.debug_line':
 NOTE: This section has relocations against it, but these have NOT been applied to this dump.
  0x00000000 d5000000 0400ad00 00000801 01fb0e0d ................
  0x00000010 00010101 01000000 01000001 2f525553 ............/RUS
  0x00000020 545f5352 432f7372 632f6270 66002f75 T_SRC/src/bpf./u
  0x00000030 73722f69 6e636c75 64652f6c 696e7578 sr/include/linux
  0x00000040 002f7573 722f696e 636c7564 652f6173 ./usr/include/as
  0x00000050 6d2d6765 6e657269 63002f4c 49424250 m-generic./LIBBP
  0x00000060 465f494e 434c5544 455f4845 41444552 F_INCLUDE_HEADER
  0x00000070 532f6270 66000075 64705f72 6f757465 S/bpf..udp_route
  0x00000080 2e627066 2e630001 00006270 662e6800 .bpf.c....bpf.h.
  0x00000090 02000069 6e742d6c 6c36342e 68000300 ...int-ll64.h...
  0x000000a0 00627066 5f68656c 7065725f 64656673 .bpf_helper_defs
  0x000000b0 2e680004 00000000 09020000 00000000 .h..............
  0x000000c0 00000309 01050f0a 21060375 20050206 ........!..u ...
  0x000000d0 030c2e4b 02020001 01                ...K.....

[anakryiko]

@insearchoflosttime, @jyn514 should we instead of these manipulations try doing BPF static linking which strips out DWARF info as sort of a side-effect. This both should solve this issue, I think, but also will make it possible to have multi-file BPF programs linked together. Kernel selftests has been doing static linking for a while now, I think it's a pretty safe thing to do.

Thoughts?

[anakryiko]

For reference, here are public libbpf APIs that we can use directly (no need for bpftool dependency): https://github.com/libbpf/libbpf/blob/master/src/libbpf.h#L1244-L1263

[jyn514]

Sorry, I'm not sure I understand - how are the APIs related to doing static linking? I probably won't have a ton of extra time to dedicate to redoing the whole PR.

(but in general I like the idea of never adding this info in the first place instead of stripping it out after the fact :) )

[insearchoflosttime]

Sorry, I'm not sure I understand - how are the APIs related to doing static linking? I probably won't have a ton of extra time to dedicate to redoing the whole PR.

I think this patchset explains more in-depth how those APIs relate to static linking. No worries if you don't have extra bandwidth on this, thanks for looking into this problem in the first place. Let us know if you'd like someone to take it over.

[jyn514]

Let us know if you'd like someone to take it over.

That would be great, thanks!

[jyn514]

@insearchoflosttime given that it seems #177 will take some time, does it make sense to merge this in the meantime to avoid the non-determinism? The code here can be removed once libbpf starts using static linking.

[insearchoflosttime]

@jyn514 Yes, I would be happy to merge this in as a temporary workaround! Can you resubmit this? Because the "deterministic-objs" branch has been deleted, I can't reopen this. Thanks!

[jyn514]

@insearchoflosttime sure thing, done :) sorry for the delay, been very busy with life.

[insearchoflosttime]

@jyn514 Do you mind rebasing and resolving merge conflicts? Thanks!

[insearchoflosttime]

~~On further thought, @jyn514 I'll pick up https://github.com/libbpf/libbpf-rs/issues/177 next week to get that in (or if you or someone else would like to work on it, that'd be awesome, but no pressure)~~

Didn't get to this this week :/ I'm going on vacation next week, so I'll try to pick this up next next week when I get back (if anyone has extra bandwidth, feel free to pick it up!)

[danielocfb]

Based on https://github.com/libbpf/libbpf-rs/issues/163#issuecomment-1006820411 and https://github.com/libbpf/libbpf-rs/pull/169#issuecomment-1205832705, I am drawing the conclusion that this pull request seems no longer applicable. Please let me know if that's not the case. Closing.

[jyn514]

@danielocfb I don't think @insearchoflosttime ever had time to work on the static linking, unfortunately. If you don't mind reopening the PR I have time to fix the conflicts :)

[danielocfb]

@danielocfb I don't think @insearchoflosttime ever had time to work on the static linking, unfortunately. If you don't mind reopening the PR I have time to fix the conflicts :)

I thought we agreed to change the approach and didn't want to go with what is done here? That is my reading of https://github.com/libbpf/libbpf-rs/issues/163#issuecomment-1006820411 anyway

[jyn514]

@danielocfb when we use static linking, we'll be able to remove -ffile-prefix-map, but I'm not sure how to get static linking working, and this avoids the non-determinism in the meantime.

[danielocfb]

@danielocfb when we use static linking, we'll be able to remove -ffile-prefix-map, but I'm not sure how to get static linking working, and this avoids the non-determinism in the meantime.

Fair enough. I think it depends on how much work getting static linking in will be, whether we want to merge this change in the meantime. Will chat with @insearchoflosttime and see if I can get a better idea.

[insearchoflosttime]

Sorry for not getting to this sooner! Feel free to take this over @danielocfb if you have bandwidth. And thanks for your patience @jyn514!

[danielocfb]

Feel free to take this over @danielocfb if you have bandwidth.

Sounds good, thanks for the update. Will take a stab at adding static linking support and then we can see if that is sufficient, in which case we may be able to close this pull request.

[danielocfb]

FWIW, I finished the implementation here. Feel free to give it a swirl and see if it resolves the problem for you, @jyn514

[jyn514]

@danielocfb unfortunately I no longer have access to the closed-source codebase I was using to test the determinism, but https://github.com/libbpf/libbpf-rs/pull/169#discussion_r778335303 is the script I was using - it should be compatible with anything using cargo libbpf make.

That said, the test you added in https://github.com/libbpf/libbpf-rs/pull/323 looks right :)