gobra icon indicating copy to clipboard operation
gobra copied to clipboard

Parallelize CI

Open ArquintL opened this issue 10 months ago • 2 comments

This PR splits up the one big CI job we have into 7 (example) and executes 4 of them in parallel to speed up our CI runs (~16 vs. 22 min). The main change is that this PR splits sbt test into executing GobraTests for each subdirectory in regression and executing all remaining test classes. I.e., we currently have the following jobs:

  • execute GobraTests on directory regressions/examples
  • execute GobraTests on directory regressions/features
  • execute GobraTests on directory regressions/issues
  • execute all remaining test classes To achieve this behavior, I've introduced a tag GobraTestsTag, which unfortunately has to be Java code. This tag allows us to run only test classes either with or without this tag. Furthermore, the CI automatically detects the subfolders in regressions and spawns jobs accordingly such that we do not miss test cases if someone adds further subdirectories.

I've marked this PR as a draft as we can discuss the following points / decisions:

  • [ ] We currently used sbt test and use now sbt testOnly to execute the test cases (instead of using a precompiled JAR file) which involves compiling the test classes in each of the 4 jobs mentioned above, which adds some overhead. Do we want to stick to this? I'm currently testing the alternative on this branch
  • [ ] The execution time varies quite a bit between these 4 jobs (~6.5, 8.25, 3.5, and 4.75min). Should we split differently? If so how? Or should we further partition regression/features?
  • [ ] Should we try to further parallelize executing the tests within each job by instructing sbt to do so?

ArquintL avatar Mar 13 '25 08:03 ArquintL

before we move on with merging this, it would be cool to understand what problems we get when enabling parallel tests in sbt and see if there's a way to fix them. That might be the simplest solution

jcp19 avatar Mar 28 '25 09:03 jcp19

@jcp19 I've tried using the ParallelTestExecution trait that executes tests within a test suite in parallel on branch parallelize-GobraTests. While I can see an impressive speedup for just a few testcases, the execution time of the entire test suite significantly increases for some reason. Furthermore, I get a lot of NoClassDefFoundError errors at runtime when executing the tests via sbt. I made several changes to even be able to run the GobraTests in that branch and, thus, I'd suggest to merge them while keeping their execution sequential

ArquintL avatar Apr 04 '25 11:04 ArquintL