Improve the CI

[jooola]

May I start working on improving the CI ? Has someone already done some work to improve this ?

I would like to:

• simplify the CI pipeline
• do more stuff in parallel
• reduce the overall run time
• prevent running tasks specific to the savonet org on forks or pull requests

This involves a lot of caching and reworking the custom scripts, maybe leverage docker layer caching. Ideally the local environment and the CI environment should be as close a possible to be able to debug/fix/improve locally and only have Github action do the checking part (it annoying to debug direclty on github action).

I would also like to add some extra tools to make the dev life easier, such as the pre-commit hooks.

[toots]

Hi,

Definitely down for that!

[jooola]

The liquidsoap-full repository is deprecated, but still used in CI, is that normal ? Shouldn't we care about liquidsoap only and get deps from opam ?

[toots]

Yeah this part of the process is still pending some cleanups.

• First, there are the various Dockerfile available at: https://github.com/savonet/liquidsoap-full/tree/master/docker. These files are used to build our base docker images (see update-all.sh and build-win32.sh)
• The ci images are built in two phases, one phase for deps and one phase for a full code-build. This is done to reduce the overhead each time we want to do a code-build.
• During deps build, the packages at the root of liquidsoap-full are all added as opam pin and we do a opam depext with them, which allows to pull external dependencies
• During the code-build, there's a more hacky mechanism to checkout the latest code, build them in-place and add the resulting directory where they are installed locally in the list of packages that ocamlfind can find. See:

      export OCAMLPATH="${LIBPATH}/_build/install/default/lib:$OCAMLPATH"
      cd $p && dune clean && dune build @install && cd ..

in configure

This is to build the images that are uploaded to hub.docker.com as base images for the CI. Then, during the CI, the build script is here:

• posix: build-posix.sh
• windows: build-win32.sh

There is also a mechanism to set specific versions to our own packages, used when building a rolling release that hasn't been updated to an underlying API change. See: checkout-deps.sh

Please feel free to propose any cleanup of this. We do not need liquidsoap-full except for that and the base website files. All that has gone through years for iterative changes and could definitely be redefined properly.

[jooola]

So I've been digging the repositories and I am starting to have a glimpse of how the current organization works.

Here is a list of ideas, questions, and proposal:

• Enable stale bot, to close old issues (only on savonet/liquidsoap)

• Enable renovate bot to provide dependencies upgrade through pull requests (this will mainly help with Github action, and maybe docker images and other dependency manager) (on all the repositories unless we use the idea below to upgrade dependencies)

• Use a meta repository to share actions and tooling for all the ocaml-* repositories, and automatically sync files through PR to each repository. This can greatly help to keep a config file in sync, for example renovate config or CI workflows.

• I suggest to have pre-commit everywhere

• I suggest to have renovate bot everywhere

• Use a dedicated Website repository to remove it from liquidsoap-full. This will also allow us to enhance the website by using a better tool to generate it. The documentation needs to be generated from ocaml, but the website can be build with any other tool, and maybe I would take some time to create a easier to use website (this should greatly help the users).

• Create a reproducible build environment based on container technologies. I propose to use buildkit. With this we can cache a lot of steps easily, run the same script in both CI and locally, use multi-stage docker files to refactor all the different docker files. Buildkit also knows how to de-duplicate work by sharing intermediate images.

• To make caching work, we should never clone a remote repository inside the docker image but rather clone and then COPY the files inside. Versioning should be handled outside the build context, so if I need version x.y.z of ocaml-, I first need to check it out, and them build.

• For caching to work even better, we can prepare a intermediate image for each dependency, this will only trigger a rebuild for that dependency if it has changed.

• For the general workflow, I was wondering if you were open to start using the fork=> branch => pull request workflow. And not commit directly to the main branch ? This is mainly to keep the CI in check for the main branch, if we have too much failure in the CI, we never actually notice if something is really broken / a true positive. This will make the commit history easier to read as there will be less "small fix" commit, because the Pull request would have checked this before merging. That said, I don't want to mess with your workflow, I don't contribute much here, just proposing to embrace best practices.

Now some questions:

• Still didn't understand if the liquidsoap-full repository is useful ? How can I build liquidsoap using only opam to fetch the required dependencies ? Can opam fetch a particular revision from the repo directly ? From what I understand I have to clone it and run opam pin .
• The build.md.in documentation seem to be outdated, but I could really use some docs to build liquidsoap from scratch. Should I read the many dockerfiles/scripts ?
• Is it possible to set and DESTDIR when installing a dependency ? For example I want to build one for the ocaml-* lib and only copy the result to a new layer, does a DESTDIR option exist ?

FROM $BASE_IMAGE as ocaml-ffmpeg

RUN dune install --prefix /dest

FROM $BASE_IMAGE as liquidsoap-build

COPY --from=ocaml-ffmpeg /dest /
RUN echo we can now use the previously build dependency

• What is the purpose of the s3 artifacts ? Wouldn't it be better to first upload to github and them to s3 ? Coudl we disable s3 artifact for pull_requests ?

[vitoyucepi]

Enable renovate bot to provide dependencies upgrade through pull requests (this will mainly help with Github action, and maybe docker images and other dependency manager) (on all the repositories unless we use the idea below to upgrade dependencies)

I'm not sure if you'll track ocaml dependencies, but renovate has no opam support. Don't know if dependabot has, probably not.

[toots]

So I've been digging the repositories and I am starting to have a glimpse of how the current organization works.

Here is a list of ideas, questions, and proposal:

• Enable stale bot, to close old issues (only on savonet/liquidsoap)

It's enabled already!

• Enable renovate bot to provide dependencies upgrade through pull requests (this will mainly help with Github action, and maybe docker images and other dependency manager) (on all the repositories unless we use the idea below to upgrade dependencies)

Could be nice but not a high priority. Most of the sensitive dependencies are maintained by us so they move up organically when we start working on them.

• Use a meta repository to share actions and tooling for all the ocaml-* repositories, and automatically sync files through PR to each repository. This can greatly help to keep a config file in sync, for example renovate config or CI workflows.

Definitely. There's already https://github.com/savonet/build-and-test-ocaml-module

• I suggest to have pre-commit everywhere

Fine with me.

• I suggest to have renovate bot everywhere

Sure but same as above, we don't have the dependencies problem that node has, thankfully!

• Use a dedicated Website repository to remove it from liquidsoap-full. This will also allow us to enhance the website by using a better tool to generate it. The documentation needs to be generated from ocaml, but the website can be build with any other tool, and maybe I would take some time to create a easier to use website (this should greatly help the users).

That would be great.

• Create a reproducible build environment based on container technologies. I propose to use buildkit. With this we can cache a lot of steps easily, run the same script in both CI and locally, use multi-stage docker files to refactor all the different docker files. Buildkit also knows how to de-duplicate work by sharing intermediate images.
• To make caching work, we should never clone a remote repository inside the docker image but rather clone and then COPY the files inside. Versioning should be handled outside the build context, so if I need version x.y.z of ocaml-, I first need to check it out, and them build.
• For caching to work even better, we can prepare a intermediate image for each dependency, this will only trigger a rebuild for that dependency if it has changed.

Yes to all of these 3. This is the part of the CI that have nerded out the most on but I'm definitely open to improvements. One thing to keep in mind is build time. Currently, we build on amd64 only in main to save time but we do build on arm64 on all OSes and arm32 on one for releases. This is done via a single dedicated arm64 machine with a semi-hacky arm32 image running on it (happy to share details, it's not ideal).

Ideally, it would be nice to build with docker buildx emulator capabilities but this would increase the build time a lot

• For the general workflow, I was wondering if you were open to start using the fork=> branch => pull request workflow. And not commit directly to the main branch ? This is mainly to keep the CI in check for the main branch, if we have too much failure in the CI, we never actually notice if something is really broken / a true positive. This will make the commit history easier to read as there will be less "small fix" commit, because the Pull request would have checked this before merging. That said, I don't want to mess with your workflow, I don't contribute much here, just proposing to embrace best practices.

Oh man, I definitely want to enforce this discipline. We don't have need for quick CI/CD pass for emergency release push so all commits should go through a branch/PR. With the increasing complexity in the code, deploy and etc. even small commits are liable to break something these days (which is a good thing!)

Now some questions:

• Still didn't understand if the liquidsoap-full repository is useful ? How can I build liquidsoap using only opam to fetch the required dependencies ? Can opam fetch a particular revision from the repo directly ? From what I understand I have to clone it and run opam pin .

Yeah though the way opam is smart, you might be able to opam pin <git url> never tried it.

• The build.md.in documentation seem to be outdated, but I could really use some docs to build liquidsoap from scratch. Should I read the many dockerfiles/scripts ?

This is definitely the most up-to date instructions. One other thing you might get caught on is that main has switched to dune while the current rolling release is still on autoconf. I'd suggest to focus on main and leave the rolling release alone, this will greatly reduce the complexity of the rewrite.

• Is it possible to set and DESTDIR when installing a dependency ? For example I want to build one for the ocaml-* lib and only copy the result to a new layer, does a DESTDIR option exist ?

FROM $BASE_IMAGE as ocaml-ffmpeg

RUN dune install --prefix /dest

FROM $BASE_IMAGE as liquidsoap-build

COPY --from=ocaml-ffmpeg /dest /
RUN echo we can now use the previously build dependency

By default, dune install puts files here: <library folder>/_build/install/default/lib you should be able to use that I believe.

• What is the purpose of the s3 artifacts ? Wouldn't it be better to first upload to github and them to s3 ?

It is because last time I checked there was some issues with the default uploader actions when ran on arm containers. Once/when this is fixed, you can remove it and use the standard upload method.

[jooola]

Renovate bot

Enable renovate bot to provide dependencies upgrade through pull requests (this will mainly help with Github action, and maybe docker images and other dependency manager) (on all the repositories unless we use the idea below to upgrade dependencies)

Could be nice but not a high priority. Most of the sensitive dependencies are maintained by us so they move up organically when we start working on them.

My main concern is github action upgrades (not ocaml dependencies), which aren't always upgraded. Having such bot on the many (many) repositories would help a lot to keep the actions up to date.

This will also help to keep the pre-commit hooks up to date.

Meta repository

Use a meta repository to share actions and tooling for all the ocaml-* repositories, and automatically sync files through PR to each repository. This can greatly help to keep a config file in sync, for example renovate config or CI workflows.

Definitely. There's already https://github.com/savonet/build-and-test-ocaml-module

Could this repository be renamed to something more generic ? So we can also host other tools and files. The build-and-test-ocaml-module action should be part of it, in a subfolder.

Reproducible builds CI/local using Docker Builkit

• Create a reproducible build environment based on container technologies. I propose to use buildkit. With this we can cache a lot of steps easily, run the same script in both CI and locally, use multi-stage docker files to refactor all the different docker files. Buildkit also knows how to de-duplicate work by sharing intermediate images.
• To make caching work, we should never clone a remote repository inside the docker image but rather clone and then COPY the files inside. Versioning should be handled outside the build context, so if I need version x.y.z of ocaml-, I first need to check it out, and them build.
• For caching to work even better, we can prepare a intermediate image for each dependency, this will only trigger a rebuild for that dependency if it has changed.

Yes to all of these 3. This is the part of the CI that have nerded out the most on but I'm definitely open to improvements. One thing to keep in mind is build time. Currently, we build on amd64 only in main to save time but we do build on arm64 on all OSes and arm32 on one for releases. This is done via a single dedicated arm64 machine with a semi-hacky arm32 image running on it (happy to share details, it's not ideal). Ideally, it would be nice to build with docker buildx emulator capabilities but this would increase the build time a lot

We might be able to simplify this by using https://github.com/docker/setup-qemu-action and run on multiple architectures when needed.

Again, I think we can speed up those builds by doing a lot of caching. We load/save the build cache/context into a directory https://github.com/moby/buildkit#local-directory-1 and keep it across workflows using https://github.com/actions/cache. Another caching option (really powerfull) is to use the Builkit Github action cache exporter which does the above but automatically https://github.com/moby/buildkit#github-actions-cache-experimental.

We can also force a full rebuild (--no-cache) every night to make sure the cache is really up to date.

Enforce the fork=>branch=>pull request workflow (no push to the main branches)

For the general workflow, I was wondering if you were open to start using the fork=> branch => pull request workflow. And not commit directly to the main branch ? This is mainly to keep the CI in check for the main branch, if we have too much failure in the CI, we never actually notice if something is really broken / a true positive. This will make the commit history easier to read as there will be less "small fix" commit, because the Pull request would have checked this before merging. That said, I don't want to mess with your workflow, I don't contribute much here, just proposing to embrace best practices.

Oh man, I definitely want to enforce this discipline. We don't have need for quick CI/CD pass for emergency release push so all commits should go through a branch/PR. With the increasing complexity in the code, deploy and etc. even small commits are liable to break something these days (which is a good thing!)

In that case you might want to start locking the main branch and prevent pushing directly to it: https://stackoverflow.com/questions/46146491/prevent-pushing-to-master-on-github

Admins should still be able to push commits and tags though.

[toots]

I forked the github action repository and added pre-commit setup and opam repository entries for our internal packages over at: https://github.com/savonet/liquidsoap-dev-tools. I think that this could act at the new liquidsoap-full for all builds.