Proposal: Add "canary" tests

[alysbrooks]

rewrite-clj helps prevent regressions by running the test suites of several projects that include it as a dependency ("canaries"), in its libs tests. In theory, Kaocha could do the same.

In theory, Kaocha should be transparent (in the sense of running tests exactly the same as other test runners) and not risk breaking the tests we're running, so this kind of regression shouldn't happen frequently. On the other hand, because avoiding test breakage is so important, avoiding the rare cases where we do break tests would be very valuable. These kinds of tests would have been valuable when developing parallelization, for example.

[plexus]

I'm not sure the value of such an endeavor will offset the cost, our development resources are scarce, let's not introduce more things that will need constant maintenance.

[alysbrooks]

I thought of a pretty easy way to do this in an ad-hoc way: create aliases for the local copy of Kaocha in our own repositories, then create a small script that runs clojure -A:test-local -m kaocha.runner in each of these. Doing it in an automated fashion with CI would be trickier, however.

[plexus]

I think kaocha is flexible enough for this, with a little bit of custom CI scripting

• check out latest kaocha
• for each libary
  • check out library latest main
  • pushd $libary
  • clojure -Sdeps '{:deps {lambdaisland/kaocha {:local/root "../kaocha"}}' -M -m kaocha.runner ...
  • check the result of $?
  • popd