warts icon indicating copy to clipboard operation
warts copied to clipboard

gh: Allow manual running of the full test suite and erlfuzz

Open TD5 opened this issue 1 year ago • 0 comments

This change introduces two new features:

  • Comment "/run-fuzzer" on a pull request to trigger a bot which will run erlfuzz and report the results by replying on the comment thread
  • Add the "full-build-and-check" tag to a pull request to force-override CI to always run tests for all applications

The two differ in how they are expressed because I have worked on the basis that:

  • Running the fuzzer is expensive, so we probably want to try to limit that to when we know we really need it and explicitly invoke it.
  • Running tests for all apps is sometimes needed because the existing mechanism for selecting apps to test is unsound: If you modify stdlib, apps that depend on it won't be analysed, but stdlib itself will. As such, using a tag means that all subsequent test runs for that pull request will return the full signal.

This change also factors out the initial build of OTP in CI into base-build.yaml so that it can be reused elsewhere, for example, when running erlfuzz.

The mechanism of using comments to execute manual CI jobs could later be extended to add other optional or expensive jobs (e.g. eqWAlizer) so that there is an official means of running them, without requiring that they be part of the usual, automatic CI suite.

TD5 avatar May 25 '23 15:05 TD5