Splitting Debuginfo

[Gankra]

Another future killer app: properly setting the flags and running the tools to produce proper symbol/debuginfo files (similar to source maps, but for native code).

This comment covers a lot of useful details on the state of this stuff in rust.

Some quick notes:

Each platform has different formats for this:

• windows pdb
• apple dsym
• linux dwp / linux "strip"
• breakpad (platform-agnostic) sym

Different platforms have different symbol servers:

• windows symbol server (can fetch exe/dll/pdb)
  • breakpad generalizes this for .sym, see mozilla's tecken
• linux debuginfod
• apple: just ask xcode lolsob
• android: suck eggs

To generate symbol files for rust you want to:

• build a release binary
  • with opt-level=3 or w/e release is
  • with codegen-units=1
  • with LTO..?
• with debug info cranked to max
  • build with debug=2
  • enable split-debuginfo which is horribly platform specific
• if targetting breakpad: run dump_syms to get a .sym
• strip the binary?

Then the user will want to:

• grab the final debuginfo/symbol file
• get the unique build ids that map the debuginfo to the binary (platform-specific, hopefully embedded in the file, but we should still get them out?)
• upload it to a symbol server?
• package it?

[Gankra]

We generate the pdb and upload it to the release, however we don't:

• handle dsym (it's a directory so as it turns out we might need to zip it up)
• handle dwp (upstream cargo issue, dwp's aren't quite ready for primetime)
• grab ids (haven't bothered yet)

[Gankra]

dwp is blocked on https://github.com/rust-lang/cargo/pull/11572

dsyms would be nice to do before shipping...

[Gankra]

The cargo issue was fixed (obvs needs to ride the trains) but https://github.com/rust-lang/rust/issues/105991 is also an argument against using it.

[mstange]

Some additional info for macOS symbol servers: Yes, there is no symbol server scheme on macOS, but there are bunch of alternatives, including the ability to plug in a custom "find my dSYM" script, and also we can just decide on our own scheme.

The alternatives are described here: https://lldb.llvm.org/use/symbols.html

As for a macOS symbol server URL scheme, for the Mozilla symbol server I'm planning to use the following scheme: https://symbols.mozilla.org/uuid/D13BD7EC88D830C49C6BC4B34F2D9156/debuginfo.dSYM.tar.bz2

And then we can supply an off-the-shelf script for use with DBGShellCommands which supports this scheme, and also teach wholesym about it.

[Gankra]

There is some work planned to cleanup the mess of symbol formats in rustc so it's easier to just ask for the right thing. Notably a new -Csplit-debuginfo=post-link mode has been approved to resolve the issues with dwp on linux. I believe it hasn't yet been implemented yet, though.

post-link debuginfo on linux would produce the more traditional and imo superior "strip" debuginfo format, which has the same basic properties as pdb and dSYM. With this you should be able to uniformly ask for a fully stripped binary and a completely independent native symbol file that contains all the info "missing" from the binary.

All standard tools would know how to handle these, and you'd be able to do either online or offline debugging of binary/crash (with the appropriate machinery to fetch the symbols from servers).

It's possible this format can be manually "polyfilled" by cargo-dist since it is apparently a case of just using the strip cli utility properly. This would however require some non-trivial work to make sure we get everything right. Work that would preferrably be upstream in rustc. (AIUI the "strip" debuginfo format is essentially taking a fully debuginfo'd ELF binary, running strip on it, and spitting out a file that contains all the sections that were removed. So it's like an ELF executable with all the actual executable parts missing.)