pytest-assume
pytest-assume copied to clipboard
astraw38
Failing pytest assume will always XPASS a test marked xfail
Simple example:
@pytest.mark.xfail(run=True, reason="Should fail w/ XFAIL")
def test_xfail():
pytest.assume(False)
Will always comeback XPASS instead of XFAIL. This is because the xfail marker makes report.failed == XPASS, and report.skipped == XFAIL. In this plugin, we manually set report to failed when assumptions fail.
Why this been closed? I still see this problem on pytest-assume==2.2.1
Got a working counter example?
There's a test for this that is passing, so if you got one that will trigger a failure, that'd help out. Or at least some output showing that it's not working as expected.
@astraw38 Here's a working example:
@pytest.mark.xfail(run=True, reason="Should fail w/ XFAIL", strict=True)
def test_xfail():
pytest.assume(False)
Note the strict=True. Without it, test returns XFAIL, as expected.
With it, for some reason, test returns XPASS.
Quick analysis:
strict-xfail checking is done in pytest_pyfunc_call. That was the hook we originally were using to do the core logic behind assumption validation; but that was dropped in favor of pytest_runtest_call because the former wouldn't work with unittest-based tests.
So the XFAIL/XPASS is set before we even get to our hook. Initial thought about checking that outcome in particular seems somewhat hacky, but would be a decent stop-gap.
