Figure out how to distribute native PDBs

[PathogenDavid]

Because snupkg only works with portable PDBs (which are .NET-specific), we cannot use the NuGet symbol server for Mochi.PhysX.Native. As such we need to figure out our own way to distribute them.

We should also look into setting up source link for them:

• https://github.com/dotnet/sourcelink#using-source-link-in-c-projects
• https://docs.microsoft.com/en-us/windows/win32/debug/source-server-and-source-indexing

Possible solutions:

• Just distribute them in the native runtime packages
  • ✔ Simple, people will get them by default
  • ❌ Bloats the packages for something many/most people will never need (They won't even get used unless they enable mixed debugging.)
  • ❌ They end up in published builds by default
• Distribute a separate symbol package
  • ✔ Simple, people who want them can get them
  • ❌ Easy to mix incompatible versions by mistake
  • ❌ Will end up in published builds by default
• Host our own symbol server
  • ✔ Simple, people who want them can get them and do so automatically as-needed
  • ❌ Has increased costs for MochiLibraries
  • ❌ People need to know to set up the symbol server source
  • ❌ Not sure if there's a good self-hosting option for symbol servers. (Found a few on GitHub but nothing battle-tested. It kinda sounds like it's possible to host one purely with static files though?)

Hosting our own symbol server is my personal preference, but it's really unclear how easy it would be to do.

[PathogenDavid]

Misc links for self-hosted symbol sources:

• https://blog.jayway.com/2011/06/19/hosting-your-own-source-symbol-server/
• https://docs.microsoft.com/en-us/windows/win32/debug/source-server-and-source-indexing
• https://stackoverflow.com/a/29323175/636752 -- Fantastic overview over different types of symbol stores. (NVIDIA's driver symbol server appears to be a 2-tier server and hosted in an S3 bucket so that encourages my hope that you can host one statically.)

I wonder if we could host one with GitHub Pages? (Edit: Considering a single build of PhysX produces 174 MB of PDBs, I'm not sure GitHub would appreciate this. I need to look into the limits on GitHub pages again. It might be viable if we only publish PDBs for full releases and not CI builds.)

[PathogenDavid]

I quickly made a "server" using symstore as described in that Stack Overflow answer. Looks to be pretty simple if that's really all there is to it. Only major issue I see is that there's a bit of a flat file database thing going on so we'd either need access to the "old" database or write our own logic to combine databases. I do wish it was documented what all these files do, I don't see how most of them would actually interact with the debugger.

Edit: I gleaned a bit more from reading https://docs.microsoft.com/en-us/windows/win32/debug/using-symstore, although it's still not totally clear how the debugger does the search. There's not actually an index or anything, so I guess it just tries accessing the desired PDB and if it exists it exists? Does the 000Admin stuff even matter then?

I'll probably have to set up a local NGINX server with access logs enabled and point my Visual Studio install at it and see what it does. I'm not super crazy about having to host a symbol server that'll end up getting hammered by ever single PDB lookup on everyone's computers.

[PathogenDavid]

Some official documentation of interest:

• https://docs.microsoft.com/en-us/windows-hardware/drivers/debugger/symbol-store-folder-tree
• https://docs.microsoft.com/en-us/windows-hardware/drivers/debugger/http-symbol-stores

Other not official links of interest:

• https://randomascii.wordpress.com/2020/03/14/creating-a-public-symbol-server-easily/
• https://randomascii.wordpress.com/2013/03/09/symbols-the-microsoft-way/ (Mostly about workflows for Linux for people used to symbol server, but it provides some insight into how symbol servers work.)
• https://sourceware.org/elfutils/Debuginfod.html (Apparently symbol server-like functionality for Linux. Very new, no idea on adoption or stuff but might be worth looking into at some point.)
• https://randomascii.wordpress.com/2011/11/11/source-indexing-is-underused-awesomeness/ (Source indexing)

Looking at his symbol server https://randomascii-symbols.commondatastorage.googleapis.com/ it does appear that the 000Admin database is optional. (You can see this in Google's script for creating their database too https://github.com/google/UIforETW/blob/95d001ad35ca3be5e4aaf65f5afe0d235327c285/package_etw.bat#L168-L172) Really suspired pingme.txt is optional though, I would've thought that would be required for the debugger to recognize it as a multi-tier index. Maybe it just assumes that for HTTP symbol servers?

The second link in particular has lots of info on the actual structure of the symbol server. It also explained why I see a lot of binaries on people's symbol servers. (They're needed for crash dumps in certain scenarios.) So we should add those too.

Mozilla's symbol server is open source, although probably way more than we actually need. https://github.com/mozilla-services/tecken

Another interesting thing: https://github.com/getsentry/symbolicator