openrewrite
CI: Daily execution on a selection of real source codes
Currently openrewrite and its official plugins are only tested with unit tests. I think it would be really beneficial to the project if recipes were tested on a daily basis on real code.
This would allow:
- checking java compatibility. For example java 8 compatibility has already been broken, but it was detected by users, not by automated testing.
- checking that recipes generate code that still builds
- checking that recipes generate code that keep the same behavior: the unit tests of the real code should still run successfully after code transformation
My vision of how it could work is, for a set of selected real open-source projects:
- checkout external project source code
- configure tooling (jdk version, maven or gradle)
- setup openrewrite maven/gradle plugin
- configure plugin to execute all available recipes that don't require parameters (or a predefined list)
- execute rewrite on project
- launch build and tests on transformed source code
The selection of external projects should be based on their characteristics (maven or gradle build, java version, ...).
NB: In a certain way, this is a bit similar to https://github.com/hipster-labs/jhipster-daily-builds
Should this be Daily or as a standalone check with every PR? (as a set of separate GH action checks)
If there are no changes, we shouldn't schedule any run
These checks often take a lot of time, so running on every PR (change) is probably not a good idea.
These checks often take a lot of time, so running on every PR (change) is probably not a good idea.
If we do a side job, we'll need to monitor it very often(or receive alerts) to avoid regressions
Maybe we can still have a small CI tasks to do a "smoke test" such as the check for compatibility with Java 8
At Moderne we've since implemented internal monitoring, where we run recipes daily on a large collection of projects, with alerting, to ensure there's no (or at least fewer) unnoticed regressions. Thanks for the suggestion!
