Build a release tool to control the release process

[sayanchowdhury]

Build a release tool to control the release process

The release has mentioned in the roadmap issue is a combination of multiple steps.

It would be beneficial to build a small release tool to coordinate the steps. The idea would be to make the interface simple enough for other members of the release team to operate and to make it in a modular manner to extend it as needed.

The initial planned items that the release tool should be capable of doing:

• Spawning the release process.
• Generate the Go/No-Go document end to end which means it should be capable to interact with changelog-aggregator and other tools
• Complete the final release steps such as interacting with origin to move the artefacts, update SDK, CF templates update etc

This is part of the roadmap item #632

Blocked on https://github.com/flatcar-linux/Flatcar/issues/838

[jepio]

Could this be a part of mantle?

[jferr]

I'm posting this here because I haven't found an official channel/place to ask this

Our company has the need to know in advance when a flatcar update (stable channel) is arriving. We are implementing automation (our needs are greater than what the fluo operator provides so we have a homegrown solution that we are developing/rolling out) and our on/offshore staff needs advance notice to ensure zero or minimal down time

This https://github.com/orgs/flatcar-linux/projects/7/views/8 shows a release for today (3/21) but one was not released yet for example. Can you point me somewhere for this info?

[dongsupark]

This https://github.com/orgs/flatcar-linux/projects/7/views/8 shows a release for today (3/21) but one was not released yet for example.

Yes, that is the plan. We are working on release including Stable, and it will be probably done in this week. As usual, the final release date can be a few days after the planned date, as CI tests for different cloud providers could result in extra work.

[jferr]

Thank you. Where can our team look to get at least a few hours notice before a release? It seems like releases come early in the morning (we are US mountain time zone). Is there a time window or exact time where the release happens/is available to the update_engine running on our hosts?

[pothos]

To be on your own schedule you can also deploy your own Nebraska server that syncs the latest release but you would allow updates only during certain timeframes or manually: https://github.com/kinvolk/nebraska/

[pothos]

Two more pointers without using Nebraska: https://www.flatcar.org/docs/latest/setup/releases/update-strategies/#disable-automatic-updates shows how to do manual updates on a machine with flatcar-update https://www.flatcar.org/docs/latest/setup/releases/update-strategies/#auto-updates-with-a-maintenance-window is about the reboot window in locksmith

[jferr]

thank you for all of the thorough and detailed answers

[pothos]

As a quick initial version I suggest the following:

• We implement a new "release" job that executes plume pre-release and plume release and then we also combine the existing scripts we have to generate the release notes draft (with package-diff we get the kernel versions for show-fixed-kernel-cves.py and then we can add this info to the output of show-changes: done here), do the CF template update, push the SDK container to ghcr, and do the publish steps for AWS/GCE/Azure Marketplace.
• The interaction with Origin is something we should do in a secure way without granting Jenkins workers full access, maybe with a second SSH account that only creates a flag file with the required information and then a service on Origin queries this flag file and gets active to execute the request.
• Later we also add the additional Azure image gallery upload to plume (and maybe rework plume to merge the pre-release and release steps) or directly to the job with az. As needed we can move code from the job script into plume or vice versa, staying flexible in terms of Go API usage vs invoking az but this is also true for other cloud marketplace interactions, avoiding the problem we have currently where manual actions exist outside of plume because we didn't find the time to write the Go code (or were blocked due to Go APIs).

[tormath1]

a few more notes:

• we can add a simple script to ci-automation/ for building and uploading containerized SDK and invoke it from Jenkins
• after the os/release we can kick-off the AWS marketplace update by using the ore dedicated command and the script in the documentation
• we need to move the cve-generator to a public repository for CVE differents from the kernel ones and/or merging it to the show-fixed-kernel-cves.py

[pothos]

The interaction with Origin is something we should do in a secure way without granting Jenkins workers full access, maybe with a second SSH account that only creates a flag file with the required information and then a service on Origin queries this flag file and gets active to execute the request.

In the beginning we can do it manually, I think it would be good to be flexible when this happens, be it before or after the release job runs (the release job would directly use the bincache). Edit: Actually not so easy because the release job creates the AMI txt files and those will be stored on Origin. So for now we must run the copy to Origin after the release job.

[t-lo]

@tormath1 could you please assign yourself if you're still planning to work on SDK publishing?

[tormath1]

@t-lo what do you think about this one: https://github.com/flatcar-linux/Flatcar/issues/739 ?

[t-lo]

I missed that one, thanks for the reference!

[pothos]

I can have a look at setting up a new publish job to run mantle's plume for AWS and Azure publishing and we can also do the mantle ore commands there for AWS marketplace uploads, now tracked in https://github.com/flatcar-linux/Flatcar/issues/838