ducktape
ducktape copied to clipboard
add decorator for tests that still runs test but not count result
It would be helpful to have a way to run tests that are known to be flakey but not have the test results be counted towards passing or failing in the final result. It is helpful to run flakey tests to see how often they fail in the context of all the other tests because the flakiness could be caused by the order of the test runs.
The --exclude
functionality simply skips the tests and the @ignore
decorator also does not run tests but at least mentions in the result tests not run which is helpful to have a quick summary of how many flakey tests were ignored.
In the old issue #112 there was a mention of a decorator called @ok_to_fail
that seems to imply the functionality desired. The decorator approach helps keep the expected test behaviour close to the definition of the test.
Would need reporter.py
to categorize the results of tests decorated as @ok_to_fail
. I suggest having 2 additional test status OPASS
and OFAIL
to distinguish from existing test status PASS
, FAIL
, and IGNORE
.