multicoretests
multicoretests copied to clipboard
Add (and use) a runner wrapping every test, to improve error reporting
This supersedes #205, providing a custom runner written in OCaml
Add a custom runner that allows to display the result of a test in the same way on Unix and Windows (by mapping Windows error codes to their equivalent result on Unix) It also uses GitHub CI formats when available so that test failures are referenced as such at their positions in the logs
As it is written in OCaml (rather than bash), it opens some possible improvements:
- create a summary of all timings,
- use per-test timeouts (the same timeout for all tests? or a custom timeout for some tests that we expect to run a long time?),
- ensure that the global timeout is never reached by stopping the test suite a couple of minutes before, so that the last steps (post-*) can run
I’ve added the suggested improvements because I’d like to use them to work on my cygwin branch. What I’m undecided about:
- whether a time-out should appear as a test failure (it could make sense to keep reporting the deadlocks we still see on Windows as failures rather than successes),
- whether we should set a default per-test timeout, and what should it be (this already makes it possible to set
TEST_TIMEOUTfor a given test via the correspondingdunefile).