[WIP] Build refactor

[TFSThiagoBR98]

This is some work for project cleanup

• [x] Split the Makefile
• [ ] Add mixed build type for both wasm and legacy support;
• [ ] Migrate from autotools to meson, to reduce patches and the build time by ≈25%;
• [ ] Add support for build flags (DEBUG, ENABLE_PTHREADS, ENABLE_LTO);
  • e.g.: make ENABLE_LTO=true DEBUG=true

[TheOneric]

I merged most of the small commits unrelated to the build system changes. As for the git clean change, not removing untracked files from our root repo makes a lot of sense to me, but the benefit isn't as clear to me for the submodules. Do you have an actual use for untracked files in the submodules? Stray files there may taint the licensecheck results and cause issues on commit updates if they collide with newly added files.

I'll write more regarding the build system changes at a later date, but I can already see that this commit does too many unrelated things at once.

[TFSThiagoBR98]

As for the git clean change, not removing untracked files from our root repo makes a lot of sense to me, but the benefit isn't as clear to me for the submodules. Do you have an actual use for untracked files in the submodules? Stray files there may taint the licensecheck results and cause issues on commit updates if they collide with newly added files.

I'll revert it for submodules.

I'll write more regarding the build system changes at a later date, but I can already see that this commit does too many unrelated things at once.

@TheOneric, I'll split this into multiple PR, for better review the changes.

[TheOneric]

I'll split this into multiple PR, for better review the changes.

Multiple commits in one PR is probably sufficient. If you prefer, feel free to use multiple PRs though.

[TFSThiagoBR98]

Multiple commits

Ok, i'll do that

[TheOneric]

Next, some remarks from my notes on the initial version and not contained in the current commits:

• Please do not require bash specifically if nothing specific to bash is being used. And iinm we currently do not use any bashisms in our build, so unless there’s a good reason we should probably continue to work with any POSIX shell.

• The old commit message the primary goal of the switch to Meson was to “reduce the number of patches.”. It doesn't appear those two are correlated at all:

  • The harfbuzz/00002 patch seemed superfluous regardless of which build system is used
  • According to #103 the Fontconfig fcstat patch is supposedly unneeded only after we bump our vendored Fontconfig version. If I didn't miss something, you did not bump the version but were removing the patch. I did test bumping fontconfig and removing the patch a couple of weeks ago: while it built, at runtime I was getting errors regarding fcstat, but hadn't yet have time to investigate further. Have you checked for runtime errors after you removed this patch?
  • Other than the two patches above it appears that the only patch which can be removed maybe because of meson is freetype/0001-disable-exports. In my testing removing the patch without changing to Meson, did not cause any build problems. What issues did this try to fix?

[TheOneric]

Now regarding the switch the benefits and drawbacks of switching many subprojects to Meson in the context of JSO.

There are some time-savings in Meson using Ninja as a backend and that it compared to autotools from git, only does one configuration step and likely does it quicker. However, I suspect the bulk of the speed improvement you measured comes from the hardcoded eight jobs being replaced by ninja’s default behaviour of using as many jobs as there are logical CPUs available. Eight jobs are too much for GitHub Action runners, which advertise two logical CPUs, and in local builds it may be either too much or too little. And in some cases, like FriBidi, our build always only uses one(!) thread (probably the -j 8 parameter was forgotten, something which could be avoided by using macros to avoid copy-paste). This by itself could also be easily achieved with Make, but sure using Meson also gives the other benefits listed before, so why not.

However, regarding speed there’s one thing that would be even better than using a more appropiate job count for each project separately and which is possible with Make but not Ninja (and therefore Meson if it cannot do a “mega-build” in which all projects use Meson): parallelise not only the individual (sub)project builds but also across all targets by using a jobserver. This means if one project no longer needs all the threads, another target can already be executed. I started an attempt at this a few months ago, but it seemed like (probably) the emmake wrapper interfered with the relevant jobserver arguments. I had the idea of removing all the emmakes in the Makefile and instead calling the root Makefile with emmake, but never actually did because other things came up which took precedence. If the jobserver can be made to play nicely with emmake the gains from it may possibly outperform the Ninja and configure related gains from Meson. The catch is HarfBuzz having deprecated its autotools build, so once we switch HB to Meson, it either will consume more threads than we’d like or we need to ensure HarfBuzz specifically is not run in parallel with other builds. If it doesn't, or we decide it isn't worth the effort, switching the projects which advertise Meson as a first-class buildsystem to Meson is probably a tad faster than using their Make-based buildsystem. We should still stop hardcoding -j8.

There also is a patch series (ninja-build/ninja/#1140) to add jobserver support to Ninja since almost 6 years which is still maintained and updated. The last response from a Ninja dev is from 2 years ago though, which doesn't exactly inspire confidence in this getting merged anytime soon.

Since we have multiple CMake-only, at least one autotools-only with prospective two more additional projects, a Meson-and-ninja-only “mega-build” seems unlikely to happen. Using top-level meson with experimental non-Meson subprojects will also suffer from not being able to parallelise across projects (in addition to it being experimental and the warnings from the docs).

[eli-schwartz]

Which are the cmake-only projects?

freetypeand harfbuzz both support Meson, cmake, and configure+make all at once, very dedicated of them. fontconfig and fribidi both support meson and configure+make

expat and brotli both support cmake and configure+make

libass is autotools only, but has work on porting to meson.

None of those seem to be cmake-only, and only libass itself is configure+make only.

[TheOneric]

Btw, since JSO compiles from git not from a tarball, the difference between a proper standard autotools setup (e.g. libass) and custom config+make systems (in FreeType’s case a thin but required wrapper around autools) is relevant as the configuration steps differ.

expat and brotli both support cmake and configure+make

I missed them supporting an autotools build; then it’s 2× CMake-or-autotools not CMake-only. Doesn't make much of a difference though, since both have a Make backend and are not Meson. WrapDB doesn't seem suitable as explained in another comment.

libass is autotools only, but has work on porting to meson.

That port only aims for Meson as a second-class system missing some features and e.g. X32, possibly enough for JSO though. Also not certain when this comes to a conclusion. But then still, libunibreak and libiconv also use Make-based systems and might be added in the future.

If I understood your explanation correctly, with a Meson-at-toplevel build using the experimental “external project” feature, the individual external projects and those controlled by Meson will use distinct thread pools which together can exceed the user-chosen limit. If so, using it does not seem desirable. Our options then would be about if we want cross-(sub)project parallelisation. If yes, we should avoid ninja-backed systems when possible (and take a look if HarfBuzz’s CMake system will continued to be supported or is also deprecated). If not, using Meson when provided as a first-class build-system is likely slightly faster than using a Make-based system. In both cases we would stick to GNU Make at top-level. If at some point in the future Ninja gains jobserver-client support, we can with GNU Make at the toplevel still get both Meson for the subprojects and cross-project parallelisation.

[eli-schwartz]

and take a look if HarfBuzz’s CMake system will continued to be supported or is also deprecated

Eh... it was removed once already, and then reverted and re-added with the rationale that "there is no need to rush" to remove it, to prevent people being caught off guard. They state in various PRs that they're not keeping it around forever.

If at some point in the future Ninja gains jobserver-client support, we can with GNU Make at the toplevel still get both Meson for the subprojects and cross-project parallelisation.

But you can use Kitware's ninja fork which does have jobserver-client support, but not jobserver support -- this can generally be worked around even for a ninja-first build with a simple Makefile:

all:
	@+ninja

It is true that at the rate things are going, the ninja devs will likely never merge it -- but, Kitware regularly rebases that patchset for new releases and this is e.g. where pip install ninja is sourced from. There are people very invested in that feature. Just don't bother with the one from Debian's repository in that case.