Streamline CI on GHA

[vhotspur]

Follow-up to the discussion in #325 (direct link).

Main point is to test LTS versions first.

[farquet]

For reference, the early access builds from Oracle can be found on dockerhub at openjdk:<jdk>-ea-oracle.

Example: docker pull openjdk:19-ea-oracle

[lbulej]

Also, is there a way to build the main workflow first, so that it does not have to wait for the others to finish?

[vhotspur]

As a follow up on #362.

At the moment, we build everything (all the old JDK versions and plugins) on each push (or PR).

Practically, the jobs need about 10-15 minutes each, all of them are done in about an hour. The main four jobs (general checks for formatting, latest JDK on Linux, Windows and MacOS) are usually done in 25 minutes.

If we want to cut down the time, we need to decide what to run and when.

There are several options available where to split things (i.e. when to decide which jobs to execute). Interestingly, Linux job takes about 10 minutes, while Windows or Mac need about 13 minutes (and the scheduling for these seems to be generally slower).

• Run all checks on master only (we merge only after PR, hence there is less of time pressure and it speeds-up getting results in PRs).
• Run all checks on tagged versions only (i.e. test only releases on on all JDKs). Generally this would speed-up things but it may require bisecting in the end.
• Run all checks when commit message contains something like [ci:all].
• Run all checks periodically (e.g. every week).
• Run all checks on PRs only when review was requested.
• Built JAR only once and share it between the jobs (the build alone takes about 3 minutes). This is something I have not investigated in much detail but it should be possible. We would probably build with JDK8 and test building on other JDKs optionally only?

We can also combine the above options somehow or drop the tests for old JDKs completely (IMHO we should keep them for testing releases at least).

@lbulej What makes the most sense for you as you probably use it the most?

@farquet, @axel22 From your perspective, what is the bare minimum where we want to test always (for example, would latest JDK on Linux only be sufficient)?

[farquet]

We have some benchmarks that are JDK11 only, so testing only the latest JDK would ignore those. Moreover, the fact that the JDK changes can be significant and can impact the functionality of the suite, it feels safer to test at least LTS releases we support.

So, I think we should at least run tests on JDK8, JDK11, JDK17 and latest JDK as soon as we open a PR or make a change there.

Periodic runs sound very interesting for "latest JDK" test. A weekly job running EA builds of OpenJDK would let us catch problems there since it's a moving target.

[vhotspur]

As a side note: the action gh-release allows even automation of releases. It should be possible to create release automatically when tag is pushed (and upload binary JARs to them automatically too).