ashinn
`(chibi test)` expected failure
It would be nice to be able to (conditionally) mark a test as an expected failure in (chibi test).
The situation here is that my library works properly on one Scheme implementation, but on another one test fails due to limitations of the Scheme implementation itself. I want to detect which implementation it’s running on, and let the test pass on the first implementation or mark it as an expected failure on the other.
Currently we don't have any way to indicate an expected failure unconditionally, only a way to skip tests, so we'd need to add that first, and update all the reporting.
You could then use cond-expand for the test cases, but that's a lot of duplicate code, and for cross-impl tests this may be fairly common. Any ideas on a preferred syntax?
I’d model it vaguely on the new test-skip:
(test-expect-failure <<body>>)
I’m okay doing something like:
(let ((failing-on-some-impls
(lambda ()
(test ...)
(test ...))))
(if (on-the-bad-impl?)
(test-expect-failure (failing-on-some-impls))
(failing-on-some-impls)))
Actually, there could be a current-test-expected-failures parameter like the filters and removers. Then you could do:
(parameterize ((current-test-expected-failures
(lambda (info)
(on-the-bad-impl?))))
(test ...)
(test ...))
With an environment variable like TEST_GROUP_REMOVE and TEST_GROUP_FILTER it could also be easy to mark an expected failure depending on the operating system (from a test script or CI job for that OS) too …