skippy icon indicating copy to clipboard operation
skippy copied to clipboard

Concurrent execution of JUnit 5 tests leading to incorrect coverage computations and improper TIA

Open pjmartos opened this issue 1 year ago • 0 comments

When using skippy for test impact analysis together with JUnit 5 tests, we're observing unpredictable coverage, wrong test classes being marked for execution, and test classes being skipped when they shouldn't, when either of the following happen:

  • Test classes are running in concurrent parallel mode, and/or
  • There are test methods scattered across two or more @Nested classes, or across across a @Nested class and the top-level class itself, or both

I believe this is due to the way coverage computations are being leveraged to determine the mapping of production classes vs test classes that visit them: https://github.com/skippy-io/skippy/blob/main/skippy-core/src/main/java/io/skippy/core/SkippyTestApi.java#L115,L131

Lines 115 and 131 assume that test classes run fully isolated and that the coverage computations observed since the last reset (i.e. in scope of a beforeAll callback) must undoubtedly and unambiguously correspond to production code exercised by the test class that has just finished running and for which we're running an afterAll callback.

pjmartos avatar Oct 14 '24 14:10 pjmartos