Automate Docker builds (Especially from Master, but might as well do for all the releases)

[marclaporte]

A sizable part of Cypht users install it via Docker. Thanks to @jonocodes via https://github.com/cypht-org/cypht-docker/issues/31, we now have: https://hub.docker.com/r/cypht/cypht

As of now, Docker releases (of Cypht stable releases) are manual. This is not a problem as we release stable versions every 2-3 months or so. However, for the development / testing / community process, it's causing quite a bit of friction. We need a way for community testers to get the latest Cypht from master. It could be a daily build, or even for each commit.

A nice side-effect is that will likely help us catch build bugs sooner.

Thoughts?

Thanks!

[jonocodes]

Ok, thinking out loud here....

We can use the date as a tag, but there is no reason to pollute dockerhub with tons of images. So yes, I do think the 'daily' version could be good here.

I think we should tag it 'nightly'. Not sure why, but I think that is a more commonly used/understood name? See https://www.mozilla.org/en-US/firefox/131.0a1/releasenotes/

So lets create a github workflow that does that every day. I think @wangxiaoerYah maybe can do this since he knows about github workflows. You can follow this for the tagging process: https://github.com/cypht-org/cypht/wiki/How-to-release-Cypht#manually-releasing-a-docker-image

To clarify every day an image will be built and pushed with the name 'cypht/cypht:nightly'. Thus overriding the previous day's build.

Separate items not discussed in this ticket:

1. automating minimal CI, to make sure nightly actually boots at least
2. automating production builds

[marclaporte]

I am OK with 'nightly' but 'daily' seems a bit better: https://en.wikipedia.org/wiki/Daily_build

[marclaporte]

https://github.com/cypht-org/cypht/releases/tag/v2.3.0 was released and now also on DockerHub: https://hub.docker.com/r/cypht/cypht/tags

[jonocodes]

v2.3.0 (release) was released and now also on DockerHub: hub.docker.com/r/cypht/cypht/tags

Great. Thanks @Shadow243

[neotwix]

Hello, Do you have plan to make a workflow for arm64 too ? sailfrog/cypht-docker contain one where the official not. thnaks for your work.

[rodriguezny]

Hello, Do you have plan to make a workflow for arm64 too ? sailfrog/cypht-docker contain one where the official not. thnaks for your work.

Do you mean builds for arm architectures ? Yes, we will add them too.

[neotwix]

Yes For the arm architecture. I should be Fine. Thanks

[rodriguezny]

Yes For the arm architecture. I should be Fine. Thanks

Ok, it will be added ASAP.

[marclaporte]

Ok, it will be added ASAP.

Please add manual instructions ASAP to ease testing.

[rodriguezny]

Hello, Do you have plan to make a workflow for arm64 too ? sailfrog/cypht-docker contain one where the official not. thnaks for your work.

Hello, I added a build for arm64.

Yes For the arm architecture. I should be Fine. Thanks

Ok, it will be added ASAP.

linux/arm64 added: https://hub.docker.com/r/cypht/cypht/tags

[rodriguezny]

Hello, Do you have plan to make a workflow for arm64 too ? sailfrog/cypht-docker contain one where the official not. thnaks for your work.

arm64 added https://hub.docker.com/r/cypht/cypht/tags, you can test it.

[marclaporte]

Re-opening as we don't yet have Docker builds from master. It would really smooth out our dev-test feedback loop, like here: https://github.com/cypht-org/cypht/issues/1153#issuecomment-2440240071

Also, for releases, the Docker part is manual. This is low priority as it's a manual operation that only needs to be done for stable releases (every few months).

@jonocodes: I remember you had a mental roadmap. Can you share some more wisdom?

Thanks!

[marclaporte]

ok, I found "After that we decide how to work this into a github action/automation for the next release perhaps. And how to maintain 'latest' and other non-versioned tags." here: https://github.com/cypht-org/cypht/pull/1001#issuecomment-2130022277

[marclaporte]

To clarify every day an image will be built and pushed with the name 'cypht/cypht:nightly'. Thus overriding the previous day's build.

I agree with the principle.

Reminder of our lifecycle: https://github.com/cypht-org/cypht/wiki/Lifecycle

Importantly, 2.x is supported for one year once 3.0 is released. Thinking of use cases, how about something like this?

• cypht/cypht:2.4.0
• cypht/cypht:master-daily
• cypht/cypht:3x-daily
• cypht/cypht:2x-daily
• cypht/cypht:3x-releases
• cypht/cypht:2x-releases

So users must proactively determine a specific version, or latest stable release per branch, or daily build for master and major versions.

[marclaporte]

1. automating minimal CI, to make sure nightly actually boots at least

So if tests fail, cypht/cypht:master-daily can be stuck to a few days ago. That is OK, as we'll fix it fast enough and it's less risky for users.

We already have automated tests for each merge requests before they are accepted in master. What would be different between daily build tests? Some longer tests?

[marclaporte]

@wangxiaoerYah: @SKB-TECH will now start working on this, so now is a good time for any suggestions.

Thanks!

[jonocodes]

Yes I think this sounds good. And I agree that having daily fall behind it tests break is the way to go.

I would go with this naming scheme as it seems more consistent with how others do it. However this is no real standard, so feel free not to.

cypht/cypht:2.4.0
cypht/cypht:daily
cypht/cypht:3-daily
cypht/cypht:2-daily
cypht/cypht:3
cypht/cypht:2

My one concern is '2-daily'. The issue is that eventually it will no longer be updated, but will still be called 'daily'. But this is a minor concern and can probably be cleared up with documentation.

[marclaporte]

@JohnXLivingston @mose @benoitg Any wisdom?

[JohnXLivingston]

@JohnXLivingston @mose @benoitg Any wisdom?

For what i understand, the main point of the daily build is to have some people test the most active branch. This is not meant for developpers that are backporting fixes from 3.x to 2.x. Those will not use docker to test, but their dev environment. If we have multiple daily version, i think nobody will use the old ones.

So, i think that only one daily build is enough, and I think it does not need to specify the major version in its name.

[JohnXLivingston]

Something else that could be usefull: having special tags pointing to the latest stable version.

When you have a docker environment (for example using docker-compose), you must specify the tag you want. Then, to update the software when there is a new release (for security fix for example), you have to do a docker-compose pull. So, when users are using tag as "2.4.0", the image won't update if the new version has a different tag ("2.4.1", "2.5.0", ...). Admins have to know there is a new version number, and have to change the configuration manually.

A common workaround is to have some special tags:

• latest, which points to the latest stable version (see for example nginx latest)
• have some tags like "develop" (equivalent to the "daily" we discuss here) and "production" (see peertube)
• have a tag with the exact version ("2.4.2") and another with the minor version ("2.4") that points to the last security patch version (see for example nginx 1.27)
• maybe same thing with "2" that points to the latest "2.x"

Those are just tags. Multiple tags can share the same build (no need to build X times, just build the new image, and change existing tags to point at it).

(i don't say that we must have all those tags, we just have to choose the preferred strategy)

[benoitg]

My only concern it that we should make sure not to create a naming convention ambiguity with the snapshots we want to create for tiki, which are human triggered quasi releases meant for dogfooding in production, vs automated daily builds.

We didn't iron out a naming convention either, but tiki snapshots should have the branch and date as part of their name.

Yes it's not the same project, but since it's mostly the same people, there is a very real risk of confusion.

But i'm not well versed enough on the assumptions of the various tooling to form an opinion on the exact naming convention we should use.

[jonocodes]

Something else that could be usefull: having special tags pointing to the latest stable version.

When you have a docker environment (for example using docker-compose), you must specify the tag you want. Then, to update the software when there is a new release (for security fix for example), you have to do a docker-compose pull. So, when users are using tag as "2.4.0", the image won't update if the new version has a different tag ("2.4.1", "2.5.0", ...). Admins have to know there is a new version number, and have to change the configuration manually.

A common workaround is to have some special tags:

* latest, which points to the latest stable version (see for example [nginx latest](https://hub.docker.com/_/nginx/tags?name=latest))

I do not like the use of 'latest' as it is ambiguous since it means different things to different docker systems/users. So I specifically avoid that term. In its place I think 'stable' may be appropriate. That being said, I dont encourage such use as a production user should at least know which major version they want to run. If 'stable' is pointing to 2, and there is a breaking change when it switches to 3, this will be problematic when the version gets changed out from under the user.

* have some tags like "develop" (equivalent to the "daily" we discuss here) and "production" (see [peertube](https://hub.docker.com/r/chocobozzz/peertube/tags))

Agreed. That is the intent of the above daily/nightly above. I generally think this is a case that should not come up since the included build tools make it unnecessary, but thats just me.

* have a tag with the exact version ("2.4.2") and another with the minor version ("2.4") that points to the last security patch version (see for example [nginx 1.27](https://hub.docker.com/_/nginx/tags?name=1.27))
* maybe same thing with "2" that points to the latest "2.x"

Yes. Those are the intent of the above mentioned 'cypht/cypht:2' which is would point to 'cypht/cypht:2.4.0'

[marclaporte]

This is needed to unblock this task: https://github.com/cypht-org/cypht/issues/1386#issuecomment-2506041009

[SKB-TECH]

@marclaporte , @jonocodes , @rodriguezny , @benoitg , @JohnXLivingston , @neotwix After having read all the comments, here is the synthesis even if we are already working on it:

1. Automate a minimal CI;
2. Automate the daily build of a docker Image and make it available in dockerHub from the master branch, taking into account linux/arm64 and linux/amd64 architecture.
3. Tag the image daily

If we've forgotten, please call us back.

[marclaporte]

This would help for @knightsg to test master for: https://github.com/cypht-org/cypht/issues/671

[marclaporte]

@SKB-TECH @rodriguezny @JohnXLivingston

https://hub.docker.com/r/cypht/cypht/tags seems good.

AFAICT, ARM64 is next (not a high priority) https://github.com/cypht-org/cypht/pull/1325#issuecomment-2618567656

Anything else planned?

[rodriguezny]

@SKB-TECH @rodriguezny @JohnXLivingston

https://hub.docker.com/r/cypht/cypht/tags seems good.

AFAICT, ARM64 is next (not a high priority) #1325 (comment)

Anything else planned?

Apart from arm64 support, we have also to automate builds for releases.

[rodriguezny]

@SKB-TECH @rodriguezny @JohnXLivingston https://hub.docker.com/r/cypht/cypht/tags seems good. AFAICT, ARM64 is next (not a high priority) #1325 (comment) Anything else planned?

Apart from arm64 support, we have also to automate builds for releases.

Auto-build for releases has been added by https://github.com/cypht-org/cypht/pull/1543

[SKB-TECH]

add multi-platform daily builds: https://github.com/cypht-org/cypht/pull/1608

[marclaporte]

What is the next step?