Test framework fails to report errors

[odersky]

Compiler version

3.4.1-RC1

Minimized example

Given in #19538.

Output

scala3-boostrapped/test passes.

Expectation

The tests should fail since there are two errors in neg tests that fail. They get reported, but don't show in the test summary.

[info] Test dotty.tools.dotc.CompilationTests.negAll started

Wrong number of errors encountered when compiling tests/neg/i19334.scala
expected: 2, actual: 1
Unfulfilled expectations:
tests/neg/i19334.scala:5
Unexpected errors:

-> following the errors:
 at 6: Missing parameter type

I could not infer the type of the parameter _$1
in expanded function:
  _$1 => f(_$1)
Expected type for the whole anonymous function:
  T

where:    T is a type variable

Wrong number of errors encountered when compiling tests/neg/i19506.scala
expected: 2, actual: 1
Unfulfilled expectations:
tests/neg/i19506.scala:8
tests/neg/i19506.scala:7
Unexpected errors:

-> following the errors:
 at 0: 3.4-migration, is not a valid choice for -source
Output from 'tests/neg/i19470.scala' did not match check file. Actual output:
-- [E007] Type Mismatch Error: tests/neg/i19470.scala:9:12 -------------------------------------------------------------
9 |    List(foo(f())) // error
         ^^^^^^^^
         Found:    Inv[? >: IO <: box IO^{f}]
         Required: box Inv[box IO^?]^?

 longer explanation available when compiling with `-explain`


Test output dumped in: tests/neg/i19470.check.out
  See diff of the checkfile (`--color=always` for colored diff)
    > git diff --no-index -- tests/neg/i19470.check tests/neg/i19470.check.out
  Replace checkfile with current output
    > mv tests/neg/i19470.check.out tests/neg/i19470.check

I also noted that in the output on the terminal error counts oscillate for other kinds of tests as well. I.e. pickling would jump from 0 to 1 and back. But in the end nothing gets reported.

It looks like a bunch of race conditions that were introduced recently to the testing framework. I am very surprised that I am the first to notice this.

This is a serious problem that needs to be treated at the highest priority. Without a reliable testing framework we cannot make progress anywhere.