reframe icon indicating copy to clipboard operation
reframe copied to clipboard

[feat] Add support for marking tests as expected failures

Open vkarak opened this issue 5 months ago • 1 comments

The main elements introduced by this PR are:

  1. The @xfail decorator to mark a test as an expected sanity failure.
  2. The xfail() builtin to mark reference tuples as expected performance failures.

Two new test states are introduced which are also logged: XFAIL for expected failures and XPASS for unexpected passes.

Design-wise the concept is the same as the one explained in #2378.

Implementation-wise there are two major changes:

  1. The execution policies code is refactored by eliminating code duplication between the serial and asynchronous policies.
  2. A new utility function reference_bounds() is introduced which is used to validate and calculate the absolute lower/upper bounds. This used to be internal to assert_reference() and now it's made public (maybe the docs there need a slight reorganization).
  3. The treatment of performance failures is rewritten as now we have to handle much more complex cases, were the performance variables of a test can simultaneously be pass, fail, xpass or xfail.

For more details on how feature works macroscopically check the tutorial and the reference docs.

Closes #2378.

vkarak avatar May 15 '25 14:05 vkarak