Release process review

[jburel]

During the recent release, a few "sticky" points were noticed making the release a bit more complicated than necessary Below is a list of points to consider with potential solutions

• Release of the various omero-* packages e.g. omero-model.

  • Possible switch to MavenCentral. Note we do not have control on when the artifacts will appear on Maven central, we do on artifactory when a package is pushed via GHA see https://github.com/ome/omero-model/blob/master/.github/workflows/gradle.yml
  • Doc job needs to be triggered manually to copy the doc (javadoc, slice2html doc) to our infra See https://ci.openmicroscopy.org/job/OMERO-DEV-release-artifacts/.

• Release of openmicroscopy

  • omero dependencies version bump. Automated see https://github.com/ome/openmicroscopy/pull/6329
  • Current generation of artifacts using https://ci.openmicroscopy.org/job/OMERO-DEV-release/ - creates the artifacts under downloads.o.org - then they are manually copied to GitHub. - Redirects are manually updated.
  • Proposal
    • creates artifacts using GHA See https://github.com/ome/openmicroscopy/pull/6318
    • No manual copy
    • Redirect to put in place if we want to keep downloads.o.org or remove that step too (long term)
    • Check: download stats from GitHub
  • https://github.com/ome/omego/pull/132 changes to retrieve artifacts from GitHub (not using redirect)

• Documentation update. The documentation release is now happening via RTD

  • Not applicable this time but generation of model glossary is manual and required a running server. Automated see https://github.com/ome/omero-documentation/pull/2258/commits/c756aa708d927b67a9a0382cf5a6f1c624475e4b
  • Dependencies version bump is currently updated manually (missed).
    • Automated via GHA: https://github.com/ome/omero-documentation/pull/2258
  • https://github.com/ome/omero-documentation#release-process was not mentioned on the release board and missed during the process. See https://github.com/ome/omero-documentation/pull/2260
  • Usage of omego for installation of server i.e. replace https://github.com/ome/omero-install/blob/develop/linux/step04_all_omero_install.sh#L15. To avoid update when switching to release like 5.7. See https://github.com/ome/omero-install/pull/266

cc @sbesson @joshmoore

[jburel]

Regarding release stats. I found that page

• https://tooomm.github.io/github-release-stats/ This could help @jrswedlow to quickly get information assuming that there is a release alongside the tag. To filter by IP etc., we will need to use the API

[joshmoore]

We would need to code something to go through all of them, but good to know that the stats are there.

[jburel]

This one is actually nicer in terms of display https://qii404.me/github-release-statistics/ Code available at https://github.com/qishibo/github-release-statistics

[sbesson]

Thanks for opening this. I see for classes of improvements. Trying to express my views and sense of priorities for each of them

1. CI & deployment pipelines: migrating to GitHub Actions as the primary tool for CI/CD is in line with the efforts that have been made in the last 12-18 months. No objections

2. Version bumps and updates: this is a natural requirement that follows the splitting of the code base into separate repositories. Anythin that reduces the manual maintenance and using either existing tools (dependabot) or custom scripts to automate update PRs following upstream releases is useful and integrates with our development worfklow. As a bonus, what would be interesting (also beyond the scope of OMERO) would be to easily generate and maintain a graph of the relationships between all these repositories.

3. Artifacts hosting: a. for all components which do not have a dedicated repository (DOckerHub, Maven Central, PyPI), GitHub releases certainly feels as the platform of choice. It has been reliably used for OMERO binaries since 12+ months and my vote is for pushing more b. for Java components, Maven Central offers well-defined advantages in terms of third-party hosting, availability to the community and integrates with services like javadoc.io which don't need to be replicate. The primary downside based on the experience with the ome-* low-level Bio-Formats components is the cost for chained release so I would not understimate the impact as this has the potential for increasing the complexity of the release process. There was a brief discussion about potentially reducing the number of server components which could also be included in this discussion and might be worth capturing

4. increase omego exposition in our tooling. Here again some mixed views

   a. For x.y.z versions, assuming we move to GitHub releases and drop the build number suffix, we are close to being able to completely predict server URLs that can easily retrieve via wget/curl, urllib3... For this use case, adding a library dependency in our system administration community to wrap such a basic command feels overkill. b. For installing major/minor/latest versions, somethign is needed to provide the layer of redirection. With binaries hosted on downloads.openmicroscopy.org, we can control this redirection via our web server proxy but there is no built-in mechanism in GitHub releases to achieve the same requirement c. part of the idea behind omego is to build some form of in-house package manager - that's definitely how it is used inhttps://github.com/ome/omero-install/pull/266. We are successfully using the library at the Ansible role level but it is fair to say the current team has limited experience in developing, designing and maintaining such software. Prior to extending and recommending it, I think we need to collectively agree that we are ready to support it moving forward.