Refactor native library compilation stack

[PJB3005]

trafficstars

The system we have for compiling native libraries for Robust is currently a mess. We need to clean this up so we can distribute natives to all platforms consistently (Fluidsynth/OpenAL Soft on macOS), distribute new libraries (SDL2) and be able to do updates to these more easily. This is currently blocking ARM support (#1443).

Background

Native libraries right now are shoddily cobbled together from a few sources. Some libraries we get from third-party NuGet packages[^1]. The rest we distribute ourselves (via our own NuGet packages we publish to nuget.org), and they are put together with a combination of "compiled ourselves" and "downloaded from the project's website"[^2].

In the latter case, https://github.com/space-wizards/build-dependencies is responsible for both containing a record of these dlls and containing the NuGet packaging files used to produce the native libraries.

When we compile natives ourselves, the scripts for doing so are stored in https://github.com/space-wizards/native-build, but they're a complete unstandardized mess.

The problem

The native libraries we get from third-party NuGets are mostly fine (well, I haven't had any issues yet).

The native libraries we publish via NuGet are a pain in the ass. It's a ton of manual work that requires doing stuff on at least three operating systems. Bonus points we keep losing the PDB files so debugging is impossible.

I would like to see the following improvements made:

• Probably merge native-build and build-dependencies so packaging of NuGets can be automated
• Mostly-automated workflows for building new versions with GitHub Actions.
• Standardize the way we compile stuff, probably with vcpkg.
• All three platforms please.
• Get rid of the "manually downloading GLFW binaries", everything from source.

[^1]: e.g. libsodium: https://www.nuget.org/packages/libsodium [^2]: e.g. GLFW: https://github.com/space-wizards/build-dependencies/tree/343d4dee05495241c3e49bebc54ee6cc4383daf9/natives/glfw/3.3

[rene-descartes2021]

Using vcpkg instead for these is a very good idea.

In a company environment I found the idea of distributing and also consuming 3rd party (compiled) native libraries via NuGet to be unacceptable. So much that when I authored such NuGet packages I improvised a way for NuGet to obtain and build such libraries from source (via nmake and MSBuild magic), with plumbing for proper include and link flags. Then vcpkg came about a couple years later- I'm happy they made that.

[rene-descartes2021]

I will tackle this. Here is proposed design:

I'll put vcpkg as a submodule at /Tools/vcpkg/. I'll design its use such that no extra manual steps beyond existing dotnet build are required, i.e. no ./vcpkg/bootstrap.sh, vcpkg integrate, or vcpkg install zstd sdl1 libsodium etc, via:

• Manually adding the vcpkg .props and .targets imports to Robust.Client.csproj.
  • Also Robust.Cliebt.Webview.csproj for CEF, but upstream doesn't have source build port yet. Building CEF from source may be not desirable, requiring 90GB storage space for the Debug build, so instead binary handling via a custom port may be possible: https://github.com/microsoft/vcpkg/issues/332
  • Or just keep using existing NuGetRobust.Natives.CEF for now.
• I'll use the manifest form of vcproj usage (vcpkg.json in project dir). This is where glfw, zstd, libsodium, etc and their versions will be listed. Will possibly copy the vcpkg .props and .targets files from the subrepo into /MSBuild/ for manual revision (or fork?) for speedup and proper paths (see the FS search for manifest line to see possible issues). Don't want this search run for each compile. Probably via consuming a property from the project which included the .props.
• The vcpkg targets run before the ClCompile target. This is what will check out the sources listed in the manifest and build them. Will improvise here as needed, using another MSBuild project as a shim if necessary.
• The vcpkg repository Readme instructs the user to run a bootstrap script. The script first tries to download a prebuilt vcpkg-tool (different repo). If none exist for the platform/arch it will download a source tar and build that. This script section can be copied out to always build from source, this script called via git hook. The resultant binary is placed at ./vcpkg within the repo.

vcpkg has ports for angle, libsodium, glfw, freetype, fluidsynth, openal-soft, zstd, sdl1, and sdl2. vcpkg can be configured to look elsewhere for ports (via vcpkg-configuration.json), for the ones space-wizards forked. Looks like there is a port for angle with patches that space-wizards uses. vcpkg does not have ports for CEF, or space-wizards/nativefiledialog.

Any problems with this design let me know. Downsides may be additional storage space(?) and additional time MSBuild uses for each compile, will try to mitigate if issue.

[moonheart08]

I don't think we ever plan to ship src instead of natives. Do not PR anything that builds libraries from source please, the entire point of the prebuilt natives is they're prebuilt.

We need a src compilation step yes but it is not to be part of the main repos, we publish the built natives to nuget for ease of building for all downstreams and game programmers.

[PJB3005]

Yes absolutely, natives need to stay prebuilt for regular engine development. The thing we're doing in the end (putting them into NuGet packages) is fine. The thing that needs fixing is how those NuGet packages are built.

Having the correct C/C++ compiler and Windows SDK available is a huge extra pile of things that can go wrong, especially for new contributors.

None of this work should change the workflow for regular developers or modify RobustToolbox itself at all, this should all stay in the other repos I linked.

[PJB3005]

And yes btw for CEF just get the downloads from Spotify's builds CDN. Absolutely not compiling that garbage from scratch even with a new compilation stack.

[AverageNotDoingAnythingEnjoyer]

Guess no-one is willing to do it, sigh