RFC: Build releases from GitHub actions

[zisoft]

The idea here is to build releases (macOS and Windows) whenever a release is ready and tagged. Just run the workflow action and the binaries can then be downloaded and put on the release page.

For that I have basically copied the nightly builds workflows and changed them to checkout the repository with a given release tag. That release tag is read from an environment variable which needs to be set in the github environment before the workflow is run:

First create the environment with the name "Release Build":

Within that environment, create the variable RELEASE_TAG and set its value to the name of the release tag:

These steps need to be done only once. In the future, only the value of the environment variable needs to reflect the current release tag.

The workflow action creates binaries for

• Windows
• macOS x86_64
• (macOS arm is announced by github for Q4/2023)

What do you think?

[parafin]

I think it's possible to implement without the need for environment variables:

https://stackoverflow.com/questions/63421008/run-a-github-action-after-new-tag-creation (read all comments)

[kmilos]

Cool, but changes the existing release process in at least two ways, which needs to analyzed and discussed:

• Today the release source tarball is packaged first, and then binaries built from that, not straight from the git source tree. (I have no opinion on this and don't really understand the consequences of not doing so.)

and, more importantly

• At least for Windows, @wpferguson builds release binaries using his MSYS2 environment that has been frozen a couple of weeks earlier and somewhat more tested, as we have been bitten by regressions or new bugs in bleeding edge versions of libraries and toolchain before. (I guess this is desirable for Homebrew as well.) Unfortunately, there is no way to pin package versions in MSYS2.

[TurboGit]

But then the source to build the release won't be signed. Today I create the tarball for the source ensuring all is ready and I sign the tarball. I'm not saying that this is blocking but that's a somewhat important change worth discussing.

[zisoft]

How about downloading the signed source tarball by the github action instead of checking out the repository? Then the URL for the tarball can be specified by an environment variable.

[parafin]

I never built from the tarball, in fact macOS release scripts won’t work correctly if not run from git. You can sign a git tag (and/or commits), you know.

[zisoft]

in fact macOS release scripts won’t work correctly if not run from git.

?? No problem here to run the (homebrew) build scripts from packaging/macosx...

[TurboGit]

You can sign a git tag (and/or commits), you know.

Yes I know, and all my release tags are signed.

[TurboGit]

How about downloading the signed source tarball by the github action instead of checking out the repository? Then the URL for the tarball can be specified by an environment variable.

Could be a solution indeed. At least this will ensure that all our versions for the three OS are built using the very same sources.

[parafin]

in fact macOS release scripts won’t work correctly if not run from git.

?? No problem here to run the (homebrew) build scripts from packaging/macosx...

Those weren’t written by me. macports scripts use git describe to get package version.

[parafin]

How about downloading the signed source tarball by the github action instead of checking out the repository? Then the URL for the tarball can be specified by an environment variable.

Could be a solution indeed. At least this will ensure that all our versions for the three OS are built using the very same sources.

IMHO the whole idea is to get rid of as many manual steps as possible, and as such it’s more logical to rely on git as the base true source, instead of some manually generated file (which would require more manual steps to run CI, than just pushing a tag). But whatever is comfortable for the people doing the work is fine.

[zisoft]

macports scripts use git describe to get package version.

ah, I see the point. It's the final step when building the package. That would indeed require some rework of the scripts when using the source tarball instead of git.

IMHO the whole idea is to get rid of as many manual steps as possible, and as such it’s more logical to rely on git as the base true source, instead of some manually generated file (which would require more manual steps to run CI, than just pushing a tag).

Exactly my thoughts. And even more important: To no longer be dependend on people running their local environments to build the release packages. What if those people are not available for whatever reason (vacation, sickness, hardware failure, ...)?

With github actions the release build is only one mouseclick away in an always reproducible manner.

[kmilos]

With github actions the release build is only one mouseclick away in an always reproducible manner.

See again my comment above - not reproducible when it comes to MSYS2 as there is no way to pin package versions. We'd have to roll our own Docker image or something instead.

[TurboGit]

Don't get me wrong I'm all for simpler & streamlined release process. I'm just pointing about what is currently done. I'm certainly not blocking (or seeing as bad) using Git tag to automatically prepare a release. My point is that we must ensure that all is under control.

[dterrahe]

See again my comment above - not reproducible when it comes to MSYS2 as there is no way to pin package versions. We'd have to roll our own Docker image or something instead.

This could work both ways (an older pinned version not containing latest bug fixes). I wonder if "we" have the capacity to take on the responsibilities of a distribution. How much testing beyond what @wpferguson has time for takes place? If packages are automatically built, it would be more clear where "our" responsibilities end. Even if this could mean a release marred by recent bugs in a distribution; the same thing will happen if Debian introduces bugs in packages we depend on. I'm quoting "we" and "our" because I'm not personally doing any of the work.

[wpferguson]

The weekly builds I put out on pixls.us really serve 2 purposes:

• get people to test the software on windows
• validate my build environment

I usually "freeze" the environment about a month before release so it gets 3 or 4 tests before we build for release.

Is there a way to "snapshot" a build machine or the list of installed packages and reuse it?

[victoryforce]

Is there a way to "snapshot" a build machine or the list of installed packages and reuse it?

The easiest solution is to download and install specific versions from https://repo.msys2.org/mingw/

The oldest versions are eventually removed, but it looks like they've been keeping versions here for about two years, so downloading specific versions from here to install in practice is pretty reliable.

[github-actions[bot]]

This pull request has been marked as stale due to inactivity for the last 60 days. It will be automatically closed in 300 days if no update occurs. Please verify it has no conflicts with the master branch and rebase if needed. Mention it now if you need help or give permission to other people to finish your work.

[github-actions[bot]]

This pull request has been marked as stale due to inactivity for the last 60 days. It will be automatically closed in 300 days if no update occurs. Please verify it has no conflicts with the master branch and rebase if needed. Mention it now if you need help or give permission to other people to finish your work.

[github-actions[bot]]

This pull request has been marked as stale due to inactivity for the last 60 days. It will be automatically closed in 300 days if no update occurs. Please verify it has no conflicts with the master branch and rebase if needed. Mention it now if you need help or give permission to other people to finish your work.

[github-actions[bot]]

This pull request has been marked as stale due to inactivity for the last 60 days. It will be automatically closed in 300 days if no update occurs. Please verify it has no conflicts with the master branch and rebase if needed. Mention it now if you need help or give permission to other people to finish your work.