gridcf
Add instructions on how to make a release
We should have instructions on the procedure to make a GCT release including:
- [x] Policy for how to decide when to make the release (ask the list?)
- [x] Tagging
- [x] Making builds for downstream distributions:
- [x] Debian
- [x] EPEL, Fedora
- [x] SUSE
- [x] Release vote for the PMC
- [x] Which files to update (autoconf files?)
@matyasselmeci Thanks for the pointers.
1. Policy for how to decide when to make the release (ask the list?)
So far we made a release only if there was a considerable amount of changes from the last release (not purely cosmetic changes!) and originally I thought about a maintenance release every quarter or so. Well, the latter didn't realize, but we also didn't have a ton of changes throughout the time, but yeah, I could have been more eager to release. ;-)
1.1 Other reasons
- security issues - then we should make a new release ASAP
- bugs - the severity of a bug controls if a new release is made or not. For the latter (i.e. non-vital bugs (labels
bug-minorandbug-normal)) the bug is fixed in the GCT source and the issue is kept open with "fixed in git master" tag until the next release is made. For major bugs (labelbug-major), i.e. core functionality is broken with a release, e.g. GridFTP is not working, MyProxy is not working, GRAM is not working, etc., a new release should be made soon. For critical bugs (labelbug-critical) a new release should be made ASAP. In case of doubt about the severity,we ask for a PMC release vote (which is cheaper than a release).
2. Tagging
In order to have updated source tarballs on https://repo.gridcf.org/gct6/ we need to create a new tag. Unfortunately that also creates a release IIRC. So immediately after creating the tag, ~one should make this new release a draft release~ (<= doesn't work any longer!) the specific release has to be deleted to make it dissappear. With a new tag, the whole toolkit is built (and tested) by Travis CI and source tarballs are copied to https://repo.gridcf.org/gct6/ (actually https://hcc-osg-repo.unl.edu/gridcf-repo/gct6/sources/ since https://github.com/gridcf/gct/commit/1e55f37e0dfa1c121e1a41489508f9dccea9c7d5) if the destination file doesn't already exist (or does exist but is damaged somehow).
3. Making builds for downstream distributions:
- Debian
- EPEL/Fedora
- SUSE
With the source tarballs updated for a new tag/proposed release we should then first check if RPMs (and maybe DEBs in the future) build correctly and successfully and if that is true, proceed with the actual release.
3.1 Debian
For Debian I think it's totally different for official packages, as versions are fixed for a specific version of the OS IIC. So newer packages only enter newer OS versions and changes/fixes/etc. would need to be backported to the version that came with the respective OS. If we create our own Debian repo (like the Globus people did), we could do the same as for SUSE packages, i.e. always provide the latest version of a package. We could actually use the OBS for building the packages and providing the corresponding repository. That way we could also provide GSI-OpenSSH and UberFTP for Debian.
3.2 EPEL/Fedora
For official EPEL/Fedora packages there is a timeframe depending on the respective distribution during which new packages are only available from special package sources so new packages can be tested before they enter the standard package sources (@ellert you know better, so correct me if I'm wrong).
RPM packages for the respective dsitribution can also be created manually using the tools @matyasselmeci created for the CI builds. See https://github.com/gridcf/gct/issues/110#issuecomment-585666603 for details.
3.3 SUSE
For packages built with the openSUSE Build Service (OBS) this is not the case, they are available as soon as they are propagated to the mirror servers.
4. Release vote for the PMC
Always propose a release vote for the PMC according to our governance document.
5. Which files to update (autoconf files?)
I think for a release no in-tree files need to be updated specifically as long as we keep major and minor version at 6 and 2 respectively, as the third component is added automatically, see https://github.com/gridcf/gct/blob/master/configure.ac#L22. I.e. the master branch should always be "releasable".
5.1 Create release page
We need to draft a release page for each new release that is not used for testing purposes (see prior releases for details).
5.2 Prepare announcement email
We need to prepare an announcement email for our [email protected] mailing list. The following template can be used:
- The subject should contain the GCT release as follows:
GCT 6.2.20201212 released
- The body should contain the raw Markdown content from the release page in
<BODY>plus the following header and footer (dates and version numbers need to be adapted for future releases!):
Dear community,
the GridCF is pleased to announce a new release of the GCT: GCT version 6.2.20201212 is a maintenance release and includes all changes since the last maintenance release in September 2019.
# GCT 6.2.20201212 (maintenance release) #
<BODY>
****
All details about this release can also be found on the corresponding GitHub releases page ([1])
[1]: https://github.com/gridcf/gct/releases/tag/v6.2.20201212
Best regards,
The GridCF
Past examples can be found in the archive of the announcement mailing list.
5.3 Update the GridCF website
We need to prepare an update for the news section on https://gridcf.org/ via PR to gridcf/gridcf.github.io.
5.4 Update the documentation
We need to update the documentation in https://github.com/gridcf/gct-docs to be in sync with the in-source documentation in https://github.com/gridcf/gct.
5.5 Create new API docs
We need to create a new version of the API docs and include it into the https://github.com/gridcf/gct-docs repo after the tagging happened. @ellert has described the creation process here, this PR shows the inclusion of new API docs:
- move created API directory into
api/dir in https://github.com/gridcf/gct-docs as6.2.<TIMESTAMP_OF_LAST_COMMIT>(e.g.6.2.1607800521for the currentv6.2.20201212release) - adapt
api/6.2symlink to point to latest API directory
@matyasselmeci @ellert @msalle : Looking forward to your thoughts.
Hi Frank, all, No strong opinions from my side. A few remarks:
- for bug fixes I would let it depend on the severity of the bug whether or not to immediately create a new release or not. A new release is considerable work, although - if I understand correctly - this is mostly on the documentation side, the rest is pretty much automated? In case of doubt we could ask for a vote (which is cheaper than a release).
- having more frequent source tarballs (and source RPMs although we'd need to decide on versioning) is probably useful, also for testing, and could be done with much less effort than a full release
@matyasselmeci, @msalle, @ellert:
So I think I have covered everything in https://github.com/gridcf/gct/issues/129#issuecomment-666484709 and included @msalle's remarks and added an implementation question for @matyasselmeci.
Could you all please go through that (again) and comment (again) as needed?
Looks fine to me. Perhaps we should come up with some standardized bug severity and/or priority levels. It seems we don't have that yet (no labels at least).
Indeed. Do you have a proposal?
For some inspiration I had a look at (Open)SUSE, RedHat, Debian: (Open)SUSE
- priority levels:
None,Low,Medium,High,UrgentandCritical (situation) - severity levels:
Enhancement,Minor,Normal,MajorandCritical. RedHat uses a somewhat easier one: - priority:
unspecified,low,medium,high,urgent - severity:
unspecified,low,medium,high,urgentDebian, just looking at the severity levels here (they also define a long list of tags which relate more to priority, see https://www.debian.org/Bugs/Developer#severities): - severity:
critical,grave,serious,important,normal,minor,wishlistI also had a look at the labels for e.g. openssl, which didn't help much further.
I think some labels like enhancement, minor, normal, major and critical would probably be more or less what we need. Major would require a speed-up with releasing, and critical would require an emergency release. All the other severities could go with the normal release.
What do you think?
@msalle
What do you think?
Let's do that! :+1: I added labels for minor, normal, major and critical bugs. But what's the difference between major and critical? I updated https://github.com/gridcf/gct/issues/129#issuecomment-666484709 (in paragraph "Other reasons") but have no idea for an example of a critical bug - other than a security issue, but these won't be handled publicly here at GitHub, so a related issue would only be created after the bug has been solved and the new release has been made in my eyes.
Not sure we'll ever need critical (certainly hope we won't) but could also be non-security. Basically something that makes a tool completely unusable. In any case, if we don't need it still doesn't harm having the label.
Hi folks, sorry I haven't commented on this yet. The whole toolkit is built and tested for each tag and each merge (or push) into master. Source tarballs are only uploaded on a tag, and each tarball is only uploaded if the destination doesn't already exist (or does exist but is damaged somehow).
It would not be very difficult to create a source tarball 'snapshot' of master in a separate directory; we could upload SRPMs as well (we make them anyway), though merges happen more frequently than taggings so I'd have to start worrying about race conditions...
We have a script to make binary RPMs using our packaging but don't run it in Travis because it would just mean compiling everything again, which takes too long.
@matyasselmeci
Hi folks, sorry I haven't commented on this yet. The whole toolkit is built and tested for each tag and each merge (or push) into master. Source tarballs are only uploaded on a tag, and each tarball is only uploaded if the destination doesn't already exist (or does exist but is damaged somehow).
Ok, that clarifies it. I'll incorporate that in https://github.com/gridcf/gct/issues/129#issuecomment-666484709.
So when someone creates a PR the toolkit is build after the PR creation and when we merge this PR the toolkit is build again. Say, we can't omit the latter in such a case, right? Because the first build (after the PR creation) happens on the remote branch or so - although the code base would be the same.
It would not be very difficult to create a source tarball 'snapshot' of master in a separate directory; we could upload SRPMs as well (we make them anyway), though merges happen more frequently than taggings so I'd have to start worrying about race conditions...
But if the uploads of snapshots would only happen for merges (merge builds would be mandatory then) and the uploads for releases would only happen for new tags we would be fine, or? If such would work, we would always have a directory with the latest source tarballs for testing purposes, so no need to create a tag for test versions or releases.
I would omit uploading the SRPMs (and maybe even the creation of SRPMs - or is this meant as a test? Then we should keep that.), as the builds for Fedora/EPEL/SUSE use and create their own SRPMs anyhow.
We have a script to make binary RPMs using our packaging but don't run it in Travis because it would just mean compiling everything again, which takes too long.
If we want to create RPMs for snapshots (not only for SUSE but also for Fedora and target distributions of EPEL, except RHEL IIRC) we could do that on the openSUSE Build Service. This creates separate RPM sources (i.e. repos) for each project (and target distribution IIC), so snapshot versions can be tested by just adding an additonal RPM source.
(BTW, thanks for writing all of this down, Frank!)
You're welcome! :-)
@matyasselmeci, @msalle, @maarten-litmaath: I updated the "how to make a release" sort of documentation in https://github.com/gridcf/gct/issues/129#issuecomment-666484709 and reordered the items (also in https://github.com/gridcf/gct/issues/129#issue-654322022) and also made the release vote a separate main point, as this should actually happen after (2) tagging and (3) downstream building (to detect any issues there) and theoretically before doing any futher stuff in (5). We can of course still prepare the actions from (5) before the release vote happens, but then have to anticipate that some of this work (e.g. API docs) might have to be redone later, if the PMC does not decide in favor of the release.