community
community copied to clipboard
knative
PROCESS CHANGE: Extend Knative Release Cycle Period
Wanted to bring up the question of changing the Knative Release cycle period. At the moment we are releasing fairly trivial releases every 6 weeks, which points to a fact that we are releasing too often for the lifecycle of the project.
There are other life-cycle stage features we might want to look at, like alpha/beta/GA releases of single release versions.
Let's start the conversation and then follow up with an AI to have someone lead writing a document to work the details out for doing less releasing, making the releases a bit more meaningful.
+1 for increasing the cycle period (both to make more meaningful releases and to lessen the burden on release leads)
I'd lean towards mirroring the k8s cycle (3 per year), but staggered by maybe a month or so to allow for the new k8s versions to be available on the cloud providers and for some of the initial kinks of a new release to get worked out
+1 and start with baby steps of 12 weeks, and iterate based on feedback increase or leave it at that.
Review that all components beyond "serving" and "eventing" have a nightly build available for testing Currently, some don't have a nightly like knative plugins (ie func, quickstart)
+1 to increasing the cycle period, but would want to be able to release more patch releases for various fixes or small changes between the longer release dates. This would not require customers to upgrade and at the same time help the us deliver a change to a customer that could be waiting for it.
for example after 1.6, we can add small changes or bug fixes (if needed) in 1.6.1 and 1.6.2 and so on.
+1
There is a lot of automation for the release process, but also a lot of manual verification of that automation is required. There's also some possible points of failure for those automations, which causes those to need to be run manually in that case. In other words, a release required a lot of human involvement and meticulous following of documentation. A payoff for this level of work doesn't seem to be there for a 6-week cycle.
May I ask what is a list of things needed to move on with this decision, or not? I remember hearing "write and circulate a blog post" in today's TOC meeting. Could someone please list all the things to be checked off to bring this to a conclusion?
+1 & +1 to all the previous comments. My preference is to start with extending the cycle as proposed to 12 weeks. Monitor feedback and eventually consider a longer cycle, but jumping directly to something like 2 releases per year seems like a big leap. As an example from Client release currently we're able to produce in average 1-2 new features + a few bug fixes in a 6 weeks cycle. That's considerably smaller release set compared to pre-1.0 times.
Per my comment during yesterday's meeting and @nader-ziada above. Let's keep the current patch release triggered at least once a week. And consider to define basic principles to cherry-picking of bug fixes and potential CVE fixes as well, refl. With extended release cycle we might be more prone to bump dependencies through patches.
In addition less frequent release might create an opportunity to improve automation around them.
Review that all components beyond "serving" and "eventing" have a nightly build available for testing Currently, some don't have a nightly like knative plugins (ie func, quickstart)
Afaik, func nightly job has been added together with auto-release config. IMO sandbox projects should be in an opt-in mode
Update: and quickstart as well. @csantanapr
https://storage.googleapis.com/knative-nightly/kn-plugin-func/latest/func_linux_amd64
https://storage.googleapis.com/knative-nightly/kn-plugin-quickstart/latest/kn-quickstart-linux-amd64
I put a bunch of comments in @upodroid 's document. If you look at the Kubernetes KEP changing their release cadence to 3x / year from 4x / year, they have some clear goals around the change in terms of documenting process and hopefully adding additional time for testing due to reduced release pressure on the various SIGs.
I'd love to get some feedback from the various consumers of Knative to balance out the contributor point of view. I'll start with two data points:
-
My personal home cluster is running
v1.0.0. This probably indicates that I'm too lazy to get on the train unless getting on the train is fully automated. I don't think there's any help for me by changing the release cadence. -
Tanzu Application Platform has patch releases quarterly. Looking at the past releases, they have been picking up the N, N-1, or occaisonally N-2 Knative release, but they couldn't possibly consume artifacts more quickly than every 13 weeks.
I don't have a horse in this race, but to echo my comment in the document:
It may seem like I'm against changing the release schedule, but I actually am excited to do it -- I just want to make sure that we bring our best arguments to the table.
+1 for docs, it would be wonderful IMHO to have more time to scope features etc that need to be documented for a specific release. This is timely for us since we're looking at revisiting some of our roadmap processes right now as well, to get more input from other WGs about docs priorities for each release.
Aside re @nader-ziada 's comment - if we are doing smaller patch releases, it would be good to have a more robust release notes process as well, to ensure that we're capturing info about patches somewhere maybe? Open to opinions on this though.
We might want to more clearly document what types of changes qualify for a cherry-pick into previous releases (per @upodroid 's document). In my mind, I'd save patch releases for:
- High priority security vulnerabilities
- High severity bugs -- those that block use of a feature without workaround, or introduce a substantial regression on the previous minor release
I don't want to introduce a lot of process around patch releases, because they should functionally be the same as the minor release they are based off.
/assign @n3wscott
Since I think he was interested in putting together a proposal
/assign
I created original doc to point out various problems around release process. I will create a bunch of docs as proposals out of it, and one of them will be the proposal to change the release cadence.
Here's the proposal for release cadence change.
FYI: Also, I've updated the original doc to include ideas for other problems I listed there. I think I will extract them to separate proposals as well.
Since we're thinking about the release cycle (and since there's a goal of loose coupling between the various knative ocmponents), would it be worth thinking about having separate releases for the each component? For example, letting serving release independently of eventing. That might help address some of the issues around releases with few impactful changes.
In theory, this could also mean that perhaps we don't need a fixed release cycle, we could only release when there are meaningful changes (which may vary by WG).
I'm sure this would take a LOT of work (and may not even be possible as things are currently set up), but figured since we're discussing changes to the release cycle may as well throw it out there.
In theory, this could also mean that perhaps we don't need a fixed release cycle; we could only release when there are meaningful changes (which may vary by WG).
-1 from my side if we still want to stay to time-boxed releases (in contrast to feature-driven releases, that tend to slip into becoming longer and longer, because what is a "meaningful change" ?). I'm a big fan of time-boxed releases as there are automatic answers to the question when to release (and you know, we as a community is really good in bikeshedding :-). Also, it's much more predictable for release leads to plan when to do the release, and then its also much less overhead to make one release including X components than to make X releases, one for each component.
And who is tracking what is the latest release? The answer is simple, there is only one latest release for everything.
I'm a big fan of the current model; please don't change that (then, I'm also a fan of a mono repo, if tooling would be better). Just increase the duration between releases to give some more time to breathe and adjust to the reality that we slowed down featurewise considerably.
@psschwei -1 from me too for feature-based releases.
@rhuss already gave all points for time-boxed releases (current model). Also, I feel such discussion is outside the scope of the current proposal. If you strongly feel the feature-driven releases are better suited for Knative, I think you should open another issue to discuss such change.
feature-based releases
My comment was less about this and more asking if, while we're making changes to the release process, do we also want to think about decoupling the release process to some degree, i.e. having distinct release processes for serving, eventing, and client. This could be something as minor as reorganizing the order we cut the repos -- all serving together, all eventing together, all client together, but all still done on the same day and at the same version. Or it could be something more substantial (each component using their own release schedule or feature based or something else...).
I don't have a strong opinion here, just thought it might be worth discussing.
I'm a big fan of the current model; please don't change that (then, I'm also a fan of a mono repo, if tooling would be better). Just increase the duration between releases to give some more time to breathe and adjust to the reality that we slowed down featurewise considerably.
+100
+1 to Just increase the duration between releases to give some more time to breathe and adjust to the reality that we slowed down featurewise considerably.
My original thought was to bump us to 12 weeks with no other changes to the process, and this would give time to change the process in other ways that are interesting and possible with more time between releases. I think there was enough interest in quarterly based releases with no other process change, the end result is a well known release dates for the project.
+1 to
Just increase the duration between releases to give some more time to breathe and adjust to the reality that we slowed down featurewise considerably.My original thought was to bump us to 12 weeks with no other changes to the process, and this would give time to change the process in other ways that are interesting and possible with more time between releases. I think there was enough interest in quarterly based releases with no other process change, the end result is a well known release dates for the project.
+1 from me as well to just bumping up the number of weeks between releases as a first step, then we can revisit what else to improve later
+1 to
Just increase the duration between releases to give some more time to breathe and adjust to the reality that we slowed down featurewise considerably.My original thought was to bump us to 12 weeks with no other changes to the process, and this would give time to change the process in other ways that are interesting and possible with more time between releases. I think there was enough interest in quarterly based releases with no other process change, the end result is a well known release dates for the project.
+1 I really liked the original proposal to simple double the time between releases without additional complexity of aligning to e.g. Kubernetes.
Looking at the current schedule for upcoming releases. I can see two possible scenarios:
- Announce the process change ASAP, given we've just released 1.7. Move 1.8 to Nov 15th.
- Keep 1.8 and 1.9 as scheduled. And extend the release cycle with 1.10. The 1.10 release would land at Christmas break Dec 27th, if I calculated that right. We will need to adjust the release date into next year anyway. It might be a good opportunity for this change.
| Release | Release Date | Serving | Eventing | PKG cut | Unpin repos |
|---|---|---|---|---|---|
| v1.8 | 2022-10-04 | nak3 | lionelvillard | 2022-09-27 | 2022-10-05 |
| v1.9 | 2022-11-15 | psschwei | evankanderson | 2022-11-08 | 2022-11-16 |
https://github.com/knative/release#release-schedule
My intention of aligning with Kubernetes was just by "happy coincidence". When we do 3 releases per year, the same as K8s, we will in most cases align with K8s without any additional constraints on us.
I really like the simplicity of cron expression @evankanderson proposed:
# minute hour day month weekday 0 0 15-21 2,6,10 2The date range 15-21 restricts the days to the 3rd week of the month; the weekday specification restricts further the day to be the day of the week which is a Tuesday within that 3rd weekof the month.
12 weeks release cycle means 52w / 12w = 4.3 releases in a year. Why doing 4? If we can do 3 and that's align us with number of releases K8s puts out in a year.
We should not attempt to match kubernetes because their goal is 3 but they have a variable length release cycle and we will never align completely. I think it would be better for our downstreams (and our contributors) to have a dependable release cadence of every quarter.
I also feel going from 6 weeks to 4 months is a big bump for everyone involved, and 6 weeks to 3 months feels closer to doubling and easier to have continued expectations on the process.
+1 for quarterly releases, I pretty much agree with what @n3wscott on this
- Announce the process change ASAP, given we've just released 1.7. Move 1.8 to Nov 15th.
- Keep 1.8 and 1.9 as scheduled. And extend the release cycle with 1.10.
Updating with 1.10 gives folks one more opportunity to get new features released before KubeCon so that probably makes the most sense.
We discussed this at the TOC meeting today, and there was general agreement with quarterly releases.
We also discussed whether to keep the 1.8 release date on Oct 4, or implement this change right away, moving the 1.8 release date to Oct 25 (unfortunately, on day 2 of Kubecon... which moves from year to year!). The general consensus seemed to be to move sooner rather than put off the changes, particularly as we hoped that the change would allow us to start further automating the process.
Team, this conflict a bit with the plans for releasing func first GA release aligning with the Knative release train. We were counting on the 1.8 release to be done by the 4th Oct, so it is ready for KubeCon NA. @lance might have more context to add here.
