pytest
pytest copied to clipboard
`xfail`ed tests are reported as `skipped` on `TestReport` instead of `passed`
- [X] a detailed description of the bug or problem you are having
- [NA] output of
pip listfrom the virtual environment you are using - [X] pytest and operating system versions
- [X] minimal example if possible
problem
when we mark a test as xfail either with strict or not, we expect to fail so in the report we should have it as outcome=passed. in the case of not strict, should always be passed or either xpass. on strict it should be fail if it didn't failed and passed or xfail otherwise
right now the only way of knowing if it was an xfail or skip is to check report.wasxfail attribute
minimal example
# test_example.py
import pytest
@pytest.mark.xfail
def test_fail():
assert False
# pytest_plugin.py
def pytest_runtest_logreport(report):
print(report)
python -m pytest -vvvv -s -p pytest_explug
❯ python -m pytest -vvvv -s -p pytest_plugin
================================= test session starts ==================================
platform darwin -- Python 3.11.7, pytest-7.4.2, pluggy-1.3.0 -- /opt/homebrew/opt/[email protected]/bin/python3.11
cachedir: .pytest_cache
rootdir: /private/tmp/testxfail
plugins: anyio-4.0.0
collected 1 item
test_example.py::test_fail
<TestReport 'test_example.py::test_fail' when='setup' outcome='passed'>
XFAIL
<TestReport 'test_example.py::test_fail' when='call' outcome='skipped'>
<TestReport 'test_example.py::test_fail' when='teardown' outcome='passed'>
reference
https://github.com/pytest-dev/pytest/blob/f74e947c1fdfef238235b7dd18c8fe52108268f2/src/_pytest/skipping.py#L266-L292
in the code above we have some statements marking the outcome as skipped and actually one (I am unaware of that flow) as passed
rep.outcome = "skipped"
# but also
rep.outcome = "passed"