pytest-dev
Wrong assumption in `_post_process_reports` discards output to terminal
When running with --html, the processing removes printing the fixtures teardown log to the terminal.
It comes from a wrong assumption that the last report status is valid for all.
https://github.com/pytest-dev/pytest-html/blob/fdcd5c6331861c338e12537313b395300007f42e/src/pytest_html/html_report.py#L279-L282
And, as the plugin modifies the test_report object it has side-effects on other users like _pytest.terminal.py.
https://github.com/pytest-dev/pytest-html/blob/fdcd5c6331861c338e12537313b395300007f42e/src/pytest_html/html_report.py#L308-L310
Testing
Example test file:
import pytest
@pytest.fixture
def config():
print('SETUP')
yield
print('TEARDOWN')
def test_example(config):
print('CALL')
assert 0
Expected output
In a normal execution, I would get the output:
python3 -m pytest --tb=short test_file.py
======================== test session starts ========================
platform linux -- Python 3.8.10, pytest-6.2.5, py-1.11.0, pluggy-1.0.0
rootdir: /home/gha/tmp/test_pytest_html
plugins: html-3.1.1, metadata-1.11.0
collected 1 item
test_file.py F [100%]
============================= FAILURES ==============================
___________________________ test_example ____________________________
test_file.py:13: in test_example
assert 0
E assert 0
----------------------- Captured stdout setup -----------------------
SETUP
----------------------- Captured stdout call ------------------------
CALL
--------------------- Captured stdout teardown ----------------------
TEARDOWN
====================== short test summary info ======================
FAILED test_file.py::test_example - assert 0
========================= 1 failed in 0.01s =========================
Real output
When running with --html it would remove the TEARDOWN output.
python3 -m pytest --tb=short --html=/tmp/out.html test_file.py
======================== test session starts ========================
platform linux -- Python 3.8.10, pytest-6.2.5, py-1.11.0, pluggy-1.0.0
rootdir: /home/gha/tmp/test_pytest_html
plugins: html-3.1.1, metadata-1.11.0
collected 1 item
test_file.py F [100%]
============================= FAILURES ==============================
___________________________ test_example ____________________________
test_file.py:13: in test_example
assert 0
E assert 0
----------------------- Captured stdout setup -----------------------
SETUP
----------------------- Captured stdout call ------------------------
CALL
------------- generated html file: file:///tmp/out.html -------------
====================== short test summary info ======================
FAILED test_file.py::test_example - assert 0
========================= 1 failed in 0.02s =========================
--- out 2022-01-19 18:04:21.025161810 +0100
+++ out_html 2022-01-19 18:04:15.325074196 +0100
@@ -15,8 +15,7 @@
SETUP
----------------------------- Captured stdout call -----------------------------
CALL
---------------------------- Captured stdout teardown ---------------------------
-TEARDOWN
+------------------ generated html file: file:///tmp/out.html -------------------
=========================== short test summary info ============================
FAILED test_file.py::test_example - assert 0
-============================== 1 failed in 0.01s ===============================
+============================== 1 failed in 0.02s ===============================
Debugging
By adding a some print in the plugin.py file (still on the version from pypi locally).
def _post_process_reports(self):
for test_name, test_reports in self.reports.items():
import pprint
print()
pprint.pprint(test_reports)
It would show that, at least, "outcome" should not be taken from the last one.
python3 -m pytest --tb=short --html=/tmp/out.html test_file.py
=========================== test session starts ============================
platform linux -- Python 3.8.10, pytest-6.2.5, py-1.11.0, pluggy-1.0.0
rootdir: /home/gha/tmp/test_pytest_html
plugins: html-3.1.1, metadata-1.11.0
collected 1 item
test_file.py F [100%]
[<TestReport 'test_file.py::test_example' when='setup' outcome='passed'>,
<TestReport 'test_file.py::test_example' when='call' outcome='failed'>,
<TestReport 'test_file.py::test_example' when='teardown' outcome='passed'>]
================================= FAILURES =================================
_______________________________ test_example _______________________________
test_file.py:13: in test_example
assert 0
E assert 0
-------------------------- Captured stdout setup ---------------------------
SETUP
--------------------------- Captured stdout call ---------------------------
CALL
---------------- generated html file: file:///tmp/out.html -----------------
========================= short test summary info ==========================
FAILED test_file.py::test_example - assert 0
============================ 1 failed in 0.02s =============================
Possible fix
When removing the overwriting on test_report.outcome and test_report.when I would get again TEARDOWN in the output.
Not checked the html output with this.
pytest-html could create a new combined test_report object instead of modifying the one given by pytest to be sure that even if this is needed, it would not change the rest of the world behavior.
I tried adding a copy before modifying test_report and it fixes the terminal output. https://github.com/pytest-dev/pytest-html/blob/fdcd5c6331861c338e12537313b395300007f42e/src/pytest_html/html_report.py#L305-L309
# outcome on the right comes from the outcome of the various
# test_reports that make up this test case
# we are just carrying it over to the final report.
+ import copy
+ test_report = copy.copy(test_report)
test_report.outcome = report_outcome
test_report.when = "call"
Thanks for the report @cladmi, much appreciated! 🙏
This is a really good observation. We're working on the "next-gen" report and as such I'm trying to spend as little time as possible fixing bugs in the current plugin.
Are you OK with having this fixed in the next major release, or is this critical to you?
Btw, in "next-gen" leaving the report-object from pytest untouched is already handled. 😊 👍
Thank you for your answer.
I do not rely on pytest-html at my job, except myself locally, so no "need" to have it fixed. It's only the terminal output that seems impacted at the moment.
Only thing that could be problematic with "next-gen" if I plan to use it, is that I see you plan to remove 3.6 support and we may still have 3.6 running for some time. Our fault but is just the current situation.
To understand the state on your side.
If I would provide a fix for the current version with tests, would it be possible that it gets integrated and released or is that a bit out of plan? I see master being updated recently and the "next-gen" branch unmodified for some time. And in master, it seems from the code that the state is the same with "test_report" being modified. So I am missing where these new things are happening and if a version of "old-gen" would still be released.
TL;DR: If I find the time, is it worth that I try providing a fix for the sake of it, of would that be more problematic for you/useless?
Have a nice day :)
@cladmi
Regarding 3.6 support. For both the current state and next-gen, what that means in reality is there will be no updates in the form of security patches etc. Which won't be happening anyway since 3.6 is EOL. As far as I know, we're not using any code that won't run with 3.6, so you should be fine to continue using it.
We won't, by means of setup.py or pyproject.toml, stop you from running it on 3.6.
You are most welcome to provide a fix, it's very much appreciated! That also makes it easier for us to make sure it gets included in next-gen.
As far as next-gen goes, we're planning on releasing a beta-version at the end of this month. I would really appreciate it if you would consider trying it out! I'm planning on making it as easy as adding a --next-gen command line flag.
Most of the action is taking place here: https://github.com/BeyondEvil/pytest-html/tree/combined-fe-and-be (last updated yesterday 😊 ). Do note that it's still very much a WIP and the first version released will not be feature-complete by a long shot. 😅
Thanks again!
@BeyondEvil Ok thank you. It's good to know for 3.6.
I will provide a test and will try some fix. The minimum of having a failing test is the best thing to try a fix.
I take note for the new version, I registered for notification on releases on this and your repository. My usage is quite for the moment with pytest-html but will see if I find any issues or behavior changes for good or bad.
Have a great day :)