reframe
reframe copied to clipboard
[feat] Add support for marking tests as expected failures
The main elements introduced by this PR are:
- The
@xfaildecorator to mark a test as an expected sanity failure. - The
xfail()builtin to mark reference tuples as expected performance failures.
Two new test states are introduced which are also logged: XFAIL for expected failures and XPASS for unexpected passes.
Design-wise the concept is the same as the one explained in #2378.
Implementation-wise there are two major changes:
- The execution policies code is refactored by eliminating code duplication between the serial and asynchronous policies.
- A new utility function
reference_bounds()is introduced which is used to validate and calculate the absolute lower/upper bounds. This used to be internal toassert_reference()and now it's made public (maybe the docs there need a slight reorganization). - The treatment of performance failures is rewritten as now we have to handle much more complex cases, were the performance variables of a test can simultaneously be
pass,fail,xpassorxfail.
For more details on how feature works macroscopically check the tutorial and the reference docs.
Closes #2378.