Use vcpkg for Linux CI

[messmerd]

trafficstars

This PR moves the Linux CI builds away from our custom Ubuntu 20.04 images to a plain Ubuntu 20.04 runner with vcpkg.

• Now using vcpkg for multiple Linux dependencies
  • Important dependencies can now stay up-to-date rather than several years old
  • There were issues with a few vcpkg packages, so I had to use system packages (for ALSA, JACK, portaudio, fltk, libgig, lv2, lilv, ...) or other solutions (for Qt) instead. Vcpkg could not build Qt without the build runner running out of memory, but for the other dependencies, the problem was that CMake could not find them or they caused linker errors.
• Removes dependence on custom Docker images for Linux CI build
  • Uses plain Ubuntu 20.04 runner
  • Provides deps-ubuntu-20.04-gcc.txt as a living document of (almost) all required system dependencies
  • Makes getting started with LMMS development easier for new devs
• Updates multiple dependencies including:
  • Qt: 5.12.8 --> 5.15.2 (all CI builds use 5.15.x now)
  • libsndfile: 1.0.28 --> 1.2.2 (enables opus and mp3 support)
  • fluidsynth: 2.1.1 --> 2.3.5
  • libsamplerate: 0.1.9 --> 0.2.2
  • ...
• Updates qt5-x11embed submodule to fix deprecation warning which was treated as an error (https://github.com/lukas-w/qt5-x11embed/pull/6)
• Fixed unused variable warnings in SampleDecoder.cpp which were treated as errors

I will try to do MinGW builds with vcpkg next.

[messmerd]

@DomClark Do you have any ideas why the Linux build is failing on the Configure step now? It probably has to do with caching but I'm unfamiliar with how that works or how to even debug it.

Run ccache --zero-stats
Statistics zeroed
Error: Process completed with exit code 1.

[DomClark]

Restoring the cache will create the build directory, so mkdir will fail. 2> /dev/null will discard any error message, but the exit code will still be non-zero. Try using mkdir -p instead.

Edit: according to the documentation, CMake will create the build directory if it doesn't exist, so you should be able to omit mkdir altogether.

[messmerd]

@DomClark Thanks, that did the trick

[Rossmaxx]

for ALSA, JACK, portaudio, fltk, libgig,

We don't have alsa on windows so better to remove alsa from here.

other solutions (for Qt)

I believe you are using whatever script we use tor msvc builds. If not, better to switch to that way if it works.

That's what i found from description. If i find anything from the diff, I'll do a review

[Rossmaxx]

Ahh shit, i thought we are doing MinGW for a moment. I double checked the title now

[JohannesLorenz]

Two general thoughts about this:

1. Someone recently said they like that we use containers in the CI because it makes it easier for them to just pull the container and directly compile LMMS - compared to an approach where they first need to install all the deps. However, I think this is not a problem - even if we change the CI build script here, the container still remains and can be used for debugging.
2. If someone asks "How can I install LMMS on Ubuntu", with a container, this question could not have been answered easily. Now, with naming the explicit dependencies, we can now say: "Install these dependencies from vcpkg.json". The list is, of course, only valid for vcpkg, so users have to either also use vcpkg, or need to find the corresponding packages for their package manager. However, I think such problems can be solved differently - e.g. for Arch Linux, the packages themselves name their dependencies.

[tresf]

Someone recently said they like that we use containers in the CI because it makes it easier for them to just pull the container and directly compile LMMS - compared to an approach where they first need to install all the deps. However, I think this is not a problem - even if we change the CI build script here, the container still remains and can be used for debugging.

I will always advocate against a container-first build in the LMMS project the additional complexity it adds to the environment has not been well-justified.

Lukas set this up and when I reached out to him privately about this in 2020, this is what he said:

BTW, I'm really unhappy myself with how the LMMS CircleCI Docker setup you linked to turned out. I mainly wrote it to offer low execution times while working around limitations within CircleCI. I was also hoping that the Docker images would help troubleshoot CI issues locally because it could remove the need to replicate the CI environment when you can just use the Docker image. The result is a pain to maintain though and I'd happily help move away from it if I had the time.

In my experience, troubleshooting build issues in a container are about twice as hard as build issues in an interactive VM or on a native machine so they must be weighed against the complexity to setup a build environment. In the case of LMMS, our AppImage requirement is an LTS Linux version, so this makes compiling these dependencies by hand -- e.g. vcpkg -- more attractive for features, libraries or versions of libraries that aren't available through the LTS Linux package manager. I don't believe I would recommend this technique to newcomers, but if we DO eventually start to recommend new developers to use vcpkg, it may make sense to find a way to re-use the ccache or a container for this to speed-up build times.

If someone asks "How can I install LMMS on Ubuntu", with a container, this question could not have been answered easily.

I think you mean "How can I ~~install~~ build LMMS on Ubuntu", which is very well defined here: https://github.com/LMMS/lmms/wiki/dependencies-ubuntu. When packages change, we should update this. It provides a quick, reproducible environment for aspiring developers without trying to troubleshoot container-related, or container-specific build issues.

"Install these dependencies from vcpkg.json"

The dependencies are listed here: https://github.com/LMMS/lmms/wiki/Compiling#libraries. If this isn't being maintained, we should probably replace it with a link to the appropriate build file.

Whether we keep them around should depend on those that prefer them to interactive VMs or native builds and how willing they are to continue helping maintain them.

[messmerd]

Someone recently said they like that we use containers in the CI because it makes it easier for them to just pull the container and directly compile LMMS

I could update our Docker image so that it mirrors the same vcpkg + apt setup from this PR. That way people who use Docker and want it for their local builds can continue using it and have all of vcpkg's up-to-date dependencies.

@tresf One big concern I have about this PR is the cache size. We have to install a lot of APT packages and some of them take up a lot of disk space. The total cache size for the APT packages is 2.3 GB which is huge. And the cache size for the vcpkg packages is ~193 MB. We are already running low of cache space and this would only make it worse.

One potential solution is to update our Linux Docker image to use vcpkg and all of the same dependencies as this PR. This would require a copy of deps-ubuntu-20.04-gcc.txt and vcpkg.json in the lmms-ci-docker repo. Then I would simply change the container CI uses from ubuntu-20.04 back to ghcr.io/lmms/linux.gcc:20.04 keeping everything else the same. A single line change. This would effectively cache all the dependencies in the Docker image so we avoid bloating our GitHub Actions cache usage. The scripts for downloading APT and vcpkg dependencies would run as usual, but they'd see that the dependencies are already installed, and ccache should avoid caching them since they existed from the start.

Yes, this would mean going back to using a custom Docker image, BUT we should be able to switch back to plain ubuntu-20.04 any time we want with a single line change - just at the expense of a larger cache size (and needing to download/build all the dependencies the 1st time due to the cache miss). Docker would simply be our way of caching dependencies.

What do you think?

[messmerd]

I removed caching of the system packages since they were contributing the most to the high cache usage. Fortunately this didn't negatively affect the build times much and the cache size is much more reasonable now.

[Rossmaxx]

There were issues with a few vcpkg packages, so I had to use system packages (for ALSA, JACK, portaudio, fltk, libgig, lv2, lilv, ...) or other solutions (for Qt) instead.

Can't we just use qt from apt for now?

[messmerd]

No, Ubuntu 20.04 only has Qt 5.12 available through apt, so I'm using a 3rd party tool to install Qt 5.15. It's the same tool we use for the MSVC builds.

[Rossmaxx]

Ok i understand now. I believe using system qt might reduce steps. 22.04 (or 24) should contain the version anyway so I thought it would be better to defer qt upgrade to the system upgrade for now.

[messmerd]

There's no real reason to put it off when it can be done now, especially since Linux is the only platform that isn't on Qt 5.15 yet.

[Rossmaxx]

Okk then

[Rossmaxx]

Fixed unused variable warnings in SampleDecoder.cpp which were treated as errors

Would it makes sense to fix this properly?

[messmerd]

The unused variable warnings were for static variables defined in the vorbis header, and defining OV_EXCLUDE_STATIC_CALLBACKS is how you prevent that header from defining them.

[Rossmaxx]

I would like to test the working before approval. Might take some days as I'm busy with exams.

[messmerd]

Currently experiencing this linker error on the MSVC builds:

[423/1461] Linking C shared module plugins\ladspa\dc_remove_1207.dll
[424/1461] Generating D:/a/lmms/lmms/plugins/LadspaEffect/swh/ladspa/flanger_1191.c
[425/1461] Linking CXX executable lmms.exe
FAILED: lmms.exe src/lmms.lib 
C:\Windows\system32\cmd.exe /C "cd . && C:/ProgramData/chocolatey/bin/ccache.exe "C:\Program Files\CMake\bin\cmake.exe" -E vs_link_exe --msvc-ver=1929 --intdir=src\CMakeFiles\lmms.dir --rc=C:\PROGRA~2\WI3CF2~1\10\bin\100226~1.0\x64\rc.exe --mt=C:\PROGRA~2\WI3CF2~1\10\bin\100226~1.0\x64\mt.exe --manifests  -- C:\PROGRA~2\MICROS~2\2019\ENTERP~1\VC\Tools\MSVC\1429~1.301\bin\Hostx64\x64\link.exe /nologo @CMakeFiles\lmms.rsp  /out:lmms.exe /implib:src\lmms.lib /pdb:lmms.pdb /version:0.0 /machine:x64 /debug /INCREMENTAL /subsystem:windows && C:\Windows\system32\cmd.exe /C "cd /D D:\a\lmms\lmms\build\src && "C:\Program Files\PowerShell\7\pwsh.exe" -noprofile -executionpolicy Bypass -file C:/vcpkg/scripts/buildsystems/msbuild/applocal.ps1 -targetBinary D:/a/lmms/lmms/build/lmms.exe -installedDir D:/a/lmms/lmms/build/vcpkg_installed/x64-windows/bin -OutVariable out""
LINK Pass 1: command "C:\PROGRA~2\MICROS~2\2019\ENTERP~1\VC\Tools\MSVC\1429~1.301\bin\Hostx64\x64\link.exe /nologo @CMakeFiles\lmms.rsp /out:lmms.exe /implib:src\lmms.lib /pdb:lmms.pdb /version:0.0 /machine:x64 /debug /INCREMENTAL /subsystem:windows /MANIFEST /MANIFESTFILE:src\CMakeFiles\lmms.dir/intermediate.manifest src\CMakeFiles\lmms.dir/manifest.res" failed (exit code 1104) with the following output:
LINK : fatal error LNK1104: cannot open file 'lilv-0.lib'

I wonder if it would happen on master too if it ever had a cache miss and had to rebuild the vcpkg packages.

[Rossmaxx]

I think it's unrelated. Also since you encountered it, I wanna tell you that MSVC error messages are a PITA to deal with, with not enough information to fix it. If this is indeed related, i would suggest you gpt it out.

[tresf]

MSVC error messages are a PITA to deal with, with not enough information to fix it. If this is indeed related, i would suggest you gpt it out.

Because they stash lines longer than 127 characters into .rsp files.... ChatGPT won't be able to help with that.

I wonder if it would happen on master too if it ever had a cache miss and had to rebuild the vcpkg packages.

Yeah I'm confused by the CI logs... is it a bad cache that it's trying to build with?

[messmerd]

@tresf

is it a bad cache that it's trying to build with?

No, a while back I think I deleted the MSVC vcpkg cache for my fork (called vcpkg-x64-*) and now it has to rebuild the packages but fails with the lilv issue on MSVC. Even my master branch fails to build so it's not just this branch.

The error may be related to this upstream update for lilv, lv2, sord, and serd back in December: https://github.com/microsoft/vcpkg/pull/42537

I'm pretty sure if LMMS/lmms's vcpkg cache were to be deleted, the same build errors would occur over there.

[messmerd]

Unlike most of our dependencies, lilv uses Meson and doesn't provide a CMake target. We are using the .pc pkg-config file to locate it and link it.

So I think I'll try creating a FindLilv.cmake file to give it a CMake target and go from there.