CI: Build dependencies in-CI

[okuoku]

From: #107

Currently, most of our CI is handling dependencies(GMP and Oniguruma in particular) as following:

• A) Install them using official or semi-official binary packaging
  • macOS: homebrew: gmp oniguruma
  • Ubuntu: apt binaries: libgmp-dev libonig-dev
  • Cygwin: official binaries: libgmp-devel libonig-devel
• B) Install them from external sources
  • Win32: via https://github.com/okuoku/nmosh-build-prerequisites/releases/tag/v0.1.1

Approach B) ie. 2-step integration was chosen because:

• Time: Building dependencies would take 30min
• Reproducibility: Building dependency for each time would not account any reproducibility
• Reliability: Built libraries needs to be tested each time

Theoretically we could merge B) process into CI(ie. 1-step integration), it would have huge impact to dev workflow and needs some mitigation.

[okuoku]

... so, at this moment, I think dropping GMP and Oniguruma dependency entirely is more viable solution; easier to implement(than implement caching & testing inside CI), future-looking(for portability esp. wasm) and expected to be more stable.

Aside that, taking 30mins more is obviously no-go for me as Windows port maintainer; it affects how many pushes I could do in one night..

Caching built libraries into GHA would be looking promising but my feeling is mixed. Long story short, it kills build reproducibility because we need to establish good caching strategy(cache-key generation etc) and it's relatively harder than 2-step approach(EDIT: because in 2-step approach, we can use Git TAG to refer actual library binary).

[higepon]

I agree that less dependencies are valuable. I think Oniguruma may be easier than GMP. I think our goal is the same, A) have CI as solid foundation and B) Make CI easier to maintenance (less complicated).

• Caching is not very ideal.
  • (Less concerning) reproducibility because cache expires.
  • This makes the CI complicated. Myself 2 years later may not understand how it works.
• 2 steps is not very ideal
  • This makes the CI complicated.
  • This introduce external dependency hell over mono repo.

So I think my goal is avoid the 2 above if possible in long run. I'm totally fine long running CI because we don't push that much and we can test them in local. Also after the CI errors fix is totally acceptable.

[okuoku]

This introduce external dependency hell over mono repo.

2step-build keeps same level of translucency with other approaches; we can reach which version of the GMP or Oniguruma used from build log.

1. In tagged revision, we can know which revision of the vcpkg had been used: https://github.com/okuoku/nmosh-build-prerequisites/tree/v0.1.1
2. We know https://github.com/microsoft/vcpkg/tree/a1737683a6f8ec5d8673890581c2a8c55838d411 was used from the tag, so we can fetch package versions from https://github.com/microsoft/vcpkg/blob/a1737683a6f8ec5d8673890581c2a8c55838d411/ports/oniguruma/portfile.cmake or https://github.com/microsoft/vcpkg/blob/a1737683a6f8ec5d8673890581c2a8c55838d411/ports/gmp/portfile.cmake

Actually, it does what vcpkg recommends.

but generally we recommend using vcpkg as a submodule for CMake projects,

Instead of adding vcpkg as submodule of mosh repository, current CI adds it to nmosh-build-prerequisites repository and refer it with a tag against it.

We can flawlessly move nmosh-build-prerequisites workflow as separate YAML of this repository with manual-triggered workflow. But unfortunately, GitHub does not provide any viable feature to release build prerequisites separately.

I'm totally fine long running CI because we don't push that much and we can test them in local.

Building same version of external project for multiple times are considered harmful because we have to pay attention to more factors. In GitHub, GHA runner images are continuously updated and old images will be removed. Building external project for each time means we have infinite versions of binaries of external projects.

In 2-step integration, when something wrong was happened in v0.1.1 to v0.1.2 change for example, we'd just have to pay attention to just two set of binaries.

Few builds, Few binaries, Few headaches.

[higepon]

Thanks I just took a look at https://github.com/okuoku/nmosh-build-prerequisites/blob/v0.1.1/build-windesktop.cmake. It looks okay to have different package and matching it with tag.

We can flawlessly move nmosh-build-prerequisites workflow as separate YAML of this repository with manual-triggered workflow. But unfortunately, GitHub does not provide any viable feature to release build prerequisites separately.

Can we then move this to here? I personally think it's totally fine to have build prerequsites in release seperately.