Parallelize CI
This PR splits up the one big CI job we have into 7 (example) and executes 4 of them in parallel to speed up our CI runs (~16 vs. 22 min).
The main change is that this PR splits sbt test into executing GobraTests for each subdirectory in regression and executing all remaining test classes. I.e., we currently have the following jobs:
- execute GobraTests on directory
regressions/examples - execute GobraTests on directory
regressions/features - execute GobraTests on directory
regressions/issues - execute all remaining test classes
To achieve this behavior, I've introduced a tag
GobraTestsTag, which unfortunately has to be Java code. This tag allows us to run only test classes either with or without this tag. Furthermore, the CI automatically detects the subfolders inregressionsand spawns jobs accordingly such that we do not miss test cases if someone adds further subdirectories.
I've marked this PR as a draft as we can discuss the following points / decisions:
- [ ] We currently used
sbt testand use nowsbt testOnlyto execute the test cases (instead of using a precompiled JAR file) which involves compiling the test classes in each of the 4 jobs mentioned above, which adds some overhead. Do we want to stick to this? I'm currently testing the alternative on this branch - [ ] The execution time varies quite a bit between these 4 jobs (~6.5, 8.25, 3.5, and 4.75min). Should we split differently? If so how? Or should we further partition
regression/features? - [ ] Should we try to further parallelize executing the tests within each job by instructing
sbtto do so?
before we move on with merging this, it would be cool to understand what problems we get when enabling parallel tests in sbt and see if there's a way to fix them. That might be the simplest solution
@jcp19 I've tried using the ParallelTestExecution trait that executes tests within a test suite in parallel on branch parallelize-GobraTests. While I can see an impressive speedup for just a few testcases, the execution time of the entire test suite significantly increases for some reason. Furthermore, I get a lot of NoClassDefFoundError errors at runtime when executing the tests via sbt.
I made several changes to even be able to run the GobraTests in that branch and, thus, I'd suggest to merge them while keeping their execution sequential