multicoretests icon indicating copy to clipboard operation
multicoretests copied to clipboard

Add (and use) a runner wrapping every test, to improve error reporting

Open shym opened this issue 2 years ago • 1 comments

This supersedes #205, providing a custom runner written in OCaml

Add a custom runner that allows to display the result of a test in the same way on Unix and Windows (by mapping Windows error codes to their equivalent result on Unix) It also uses GitHub CI formats when available so that test failures are referenced as such at their positions in the logs

As it is written in OCaml (rather than bash), it opens some possible improvements:

  • create a summary of all timings,
  • use per-test timeouts (the same timeout for all tests? or a custom timeout for some tests that we expect to run a long time?),
  • ensure that the global timeout is never reached by stopping the test suite a couple of minutes before, so that the last steps (post-*) can run

shym avatar Feb 28 '23 15:02 shym

I’ve added the suggested improvements because I’d like to use them to work on my cygwin branch. What I’m undecided about:

  • whether a time-out should appear as a test failure (it could make sense to keep reporting the deadlocks we still see on Windows as failures rather than successes),
  • whether we should set a default per-test timeout, and what should it be (this already makes it possible to set TEST_TIMEOUT for a given test via the corresponding dune file).

shym avatar Mar 02 '23 18:03 shym