pytest icon indicating copy to clipboard operation
pytest copied to clipboard

`xfail`ed tests are reported as `skipped` on `TestReport` instead of `passed`

Open scratchmex opened this issue 1 year ago • 0 comments

  • [X] a detailed description of the bug or problem you are having
  • [NA] output of pip list from the virtual environment you are using
  • [X] pytest and operating system versions
  • [X] minimal example if possible

problem

when we mark a test as xfail either with strict or not, we expect to fail so in the report we should have it as outcome=passed. in the case of not strict, should always be passed or either xpass. on strict it should be fail if it didn't failed and passed or xfail otherwise

right now the only way of knowing if it was an xfail or skip is to check report.wasxfail attribute

minimal example

# test_example.py
import pytest

@pytest.mark.xfail
def test_fail():
    assert False
# pytest_plugin.py
def pytest_runtest_logreport(report):
    print(report)
python -m pytest -vvvv -s -p pytest_explug
❯ python -m pytest -vvvv -s -p pytest_plugin
================================= test session starts ==================================
platform darwin -- Python 3.11.7, pytest-7.4.2, pluggy-1.3.0 -- /opt/homebrew/opt/[email protected]/bin/python3.11
cachedir: .pytest_cache
rootdir: /private/tmp/testxfail
plugins: anyio-4.0.0
collected 1 item

test_example.py::test_fail 
<TestReport 'test_example.py::test_fail' when='setup' outcome='passed'>
XFAIL
<TestReport 'test_example.py::test_fail' when='call' outcome='skipped'>
<TestReport 'test_example.py::test_fail' when='teardown' outcome='passed'>

reference

https://github.com/pytest-dev/pytest/blob/f74e947c1fdfef238235b7dd18c8fe52108268f2/src/_pytest/skipping.py#L266-L292

in the code above we have some statements marking the outcome as skipped and actually one (I am unaware of that flow) as passed

rep.outcome = "skipped"
# but also
rep.outcome = "passed"

scratchmex avatar Jun 26 '24 18:06 scratchmex