Concurrent execution of JUnit 5 tests leading to incorrect coverage computations and improper TIA
When using skippy for test impact analysis together with JUnit 5 tests, we're observing unpredictable coverage, wrong test classes being marked for execution, and test classes being skipped when they shouldn't, when either of the following happen:
- Test classes are running in
concurrentparallel mode, and/or - There are test methods scattered across two or more
@Nestedclasses, or across across a@Nestedclass and the top-level class itself, or both
I believe this is due to the way coverage computations are being leveraged to determine the mapping of production classes vs test classes that visit them: https://github.com/skippy-io/skippy/blob/main/skippy-core/src/main/java/io/skippy/core/SkippyTestApi.java#L115,L131
Lines 115 and 131 assume that test classes run fully isolated and that the coverage computations observed since the last reset (i.e. in scope of a beforeAll callback) must undoubtedly and unambiguously correspond to production code exercised by the test class that has just finished running and for which we're running an afterAll callback.