Releasing

[balhar-jakub]

Current State

• We have active and maintenance release streams. For active release stream we release all the components at least every 6 weeks. For maintenance every 12 weeks.
• This is fully valid for server side.
• Client side releases more or less as they need, with synchronization for Zowe releases, that isn't necessary for them.
• For active release we then hold System demo showcasing what was released.
• We aren't really able effectively to release patch releases on the server side.
• Problems in any area may delay overall release of any other area as we deliver server-side together and as well with client side.

Problems to solve

• Limit release dependencies for projects that actually has dependencies
• We aren't effective in releasing patches for bugs and/or security fixes
• The active releases for server-side are too frequent for customers to benefit from them.
• Limit the workload around the releases on the Systems squad

[balhar-jakub]

Proposal from Jakub Balhar

• Zowe is a suite of projects. There is no need for the projects to be on the same version, even more so as this isn't the case for some of the projects anyway.
• As such we should do release of the suite once a quarter with different versions of different components. This factually splits at least the client side components from the server side components. The quarterly release is primarily for new features. The projects can and should release patch releases to fix bugs and remove vulnerabilities.
  • Discuss whether server side components should be packaged together.
  • This release isn't bound to specific projects within Zowe. E.g. We may release 3.1 that consists of API ML 3.5.1, ZLUX in 3.1.0, CLI 9.3.3 and the Zowe release means that these things should work together.
• Maintenance releases won't be regular and will be released as patch when there is a fixed bug or vulnerability.

[MarkAckert]

There is no need for the projects to be on the same version, even more so as this isn't the case for some of the projects anyway.

I'm repeating and supporting much of what you said in your comment: the internal versions of the components are separate, or can be separated easily from the official Zowe releases today. The value of the official Zowe releases bundling technologies together into a common version was to clarify their compatibility and joint use for end-users, and we should keep prioritizing that clarity. Even for the CLI, when we say we're releasing on 8.x, users are still either installing it from the official bundled release on zowe.org or via npm tag such as zowe/cli@zowe-v2-lts, both of which convey pretty clear compatibility expectations. If we want to further split component releases (e.g. one Zowe release has just CLI updates), that's fine, but with the caveat that it cannot be a major version boundary release.

The active releases for server-side are too frequent for customers to benefit from them.

I'm OK with reducing our release cadence. I've seen examples of this in other large communities, such as Kubernetes, where the reduced cadence is usually driven by a combination of (a) reducing overhead for the community in feature merging and end-of-pipe validation activities and (b) reducing overhead for consumers trying to stay current. If we're giving our end-users too many releases to keep up with, it sounds like a good idea to slow down.

I didn't see a proposed change for feature releases, so I would suggest going from twice quarterly to once quarterly as a starting point. With a schedule change, we'll need to stay on top of integrating features throughout the quarter (add a mid-quarter checkpoint on TSC?); if too many features are held to the end-of-quarter, the increased change volume could make the release process longer than before.

Maintenance releases won't be regular and will be released as patch when there is a fixed bug or vulnerability.

I think users appreciate scheduled drops of maintenance releases that they can plan around. If we only create maintenance releases on-demand, they're going to be stuck reacting on short notice or stop paying attention. I think the current quarterly cadence is OK, or semi-annualy is probably OK too. Of course we keep the option to create emergency releases in response to critical, exploitable vulnerabilities.

[balhar-jakub]

@MarkAckert Thanks for the points. Based on the notes in your text, what I would propose is following:

Releasing

• The projects within Zowe release patch and minor versions as needed.
• Zowe project releases quarterly statements within the active stream that validate that the components of specific versions work well together.
• Internally, Zowe validates integration between components at least every six weeks to limit integration risks.

Frequency

• Feature releases are available within the active release stream every quarter.
• Maintenance releases every six months unless an exploitable security vulnerability above medium severity triggers us to release.

Impacts

• Client side
  • Limited as the client side is already releasing separately.
• Server side
  • We need to analyze what it would take to release separate server-side components so we can deliver patch releases for separate Zowe server-side projects as needed. Maybe separate PTFs for separate components?

[balhar-jakub]

It would be useful to create a question of the month with this topic.

[balhar-jakub]

The discussion about integration of server side either as whole or as a separate pieces within the z/OS has major impact on this discussion.

What would change the story:

• e.g. some parts on z/OS
• Improved upgrade experience
• ...

We probably shouldn't put back the back-level fixes.

[balhar-jakub]

Another round of proposal based on the previous discussion and the discussion within the TSC. There is another requirement we need to keep in mind based on what we found recently:

• We need to release patch version to important critical vulnerabilities for any component as fast as possible.

Releasing

• The projects within Zowe release patch and minor versions as needed
  • Client-side releases are already independent
  • Server-side releases need to become independent
    • Proposal - Create new way of packaging, One FMID which will contain only the Launcher and all the other parts of the server side will be released and applied as separate PTFs
• Zowe releases every three month a quarterly version that collects specific versions of Zowe components and states that these works together well.
• Internally Zowe continues validating the integration with the latest PTFs from every component

Benefits

• The users can easily select what exactly do they want to install
• There will be stable version of Zowe
• It's going to be easier to add server side extensions with this mechanism and it will look more like the standard SMP/E procedure used within z/OS

Frequency

• Feature releases are available within the active release stream every quarter.
• Maintenance releases every six months unless an exploitable security vulnerability above medium severity triggers us to release.

[balhar-jakub]

As a follow-up to the discussion on 05/30, below are outlined the proposals for the short-term.

Short-term

Objectives

• Give the extenders time to validate the release
• Make sure the customers are able to upgrade the Zowe versions
• Limit the risks of slipping release dates

Proposal

• Feature releases once every quarter
  • Month before the release date the Code Freeze happens
• Bug fixes release once more in a quarter without feature updates
  • The same approach as at the moment.

[adam-wolfe]

I spoke with the CLI squad, and we found this proposal to be acceptable.