pytest-dev
[Feature Request] Should qtbot be able to take+save screenshots?
Often when dealing with CI systems, I would love to able to view a screenshot of what a QWidget looked like. It got me thinking, how hard is it to take a screenshot of a QWidget, and apparently it's not difficult at all, which got me wondering if this was some functionality pytest-qt should offer.
I would be happy to attempt to integrate this functionality (I've never written anything for a pytest plugin before), but before I did so, I figured I would ask what folks here thought.
What do you see as the benefit which pytest-qt integration can provide? I'm thinking there might be something related to picking filenames based on the test module/class/function (plus any parametrization) plus a counter. Maybe optionally collecting them into a zip/tarball. Possibly some integration with https://pypi.org/project/allure-pytest/ (or maybe there are other options).
@altendky primary use case I see would be to save a screen shot on a failed test.
Idea originally came to me when I saw that azure pipelines supports attachments to tests (however i don't think this can be done with junit yet). Having a record of passing screen shots can also be useful for determining if you have sufficient testing criteria.
primary use case I see would be to save a screen shot on a failed test.
This seems like an interesting idea, even perhaps saving an screenshot of the entire desktop (QDesktopWidget.grab I think).
@altendky has a good point though, what should be done with the screenshot at that point? Unfortunately every CI has its own way to publish test attachments, so it is kind of CI specific...
Hmmm I wonder though if we can use pytest-html support to attach images to test reports instead? Then if a user configures to publish the HTML report somewhere, the images would be available too, which seems like an excellent integration option. 🤔
If just a widget, which one? I guess maybe just whatever you directly register with qtbot? And how do we hook this while the widgets are still in scope without just making the coder add a call to every test?
On the Azure point, https://pypi.org/project/pytest-azurepipelines/. Though I can't say I've looked at what it supports or how it works.
If just a widget, which one?
I think the entire desktop can be useful.
And how do we hook this while the widgets are still in scope without just making the coder add a call to every test?
It should be possible to do that during the test report stage.
primary use case I see would be to save a screen shot on a failed test.
This seems like an interesting idea, even perhaps saving an screenshot of the entire desktop (
QDesktopWidget.grabI think).@altendky has a good point though, what should be done with the screenshot at that point? Unfortunately every CI has its own way to publish test attachments, so it is kind of CI specific...
Hmmm I wonder though if we can use
pytest-htmlsupport to attach images to test reports instead? Then if a user configures to publish the HTML report somewhere, the images would be available too, which seems like an excellent integration option. 🤔
In terms of what should be done with the screenshot following, I would argue that falls outside the scope for pytest-qt. Ideally you can specify a (relative) path, and it will put the image there, it will be up to the user to relocate that file to an artifact area, or an attachment area, or whatever their CI system supports.
On the Azure point, https://pypi.org/project/pytest-azurepipelines/. Though I can't say I've looked at what it supports or how it works.
I used pytest-azurepipelines, it's a bit outdated, as it doesn't support uploading coverage reports to their fancy coverage report tab (and the way it hijacks your pytest arguments, it makes it impossible to try and upload the coverage report yourself), but more importantly, since azure-pipelines doesn't support attachments for junit test configurations presently, so as far as I know, there is no way with pytest to get those attachments into the test report tool, but even if there was, that would fall well outside the scope of this plugin. I was merely sharing the scenario of which I had this idea come about.
Does azure support uploading generic HTML reports attached to a build?
Azure supports general artifact attachment of whatever you want (including html files). To integrate nicely into their report interface, they support a variety of test result formats. They do support coverage html reports when accompanied with a summery coverage.xml file.
Where the killer feature is if we could incorporate screenshots into the azure test attachments. As of right now, they only support that for Visual Studio Test and NUnit 3 test formats, and not junit.
I should add that I would have use for this in not just CI systems but even when running locally via tox, being able to see if different versions of Qt result in widgets (or overall screen shot) looking different would be nice.
@nicoddemus the "test report stage" occurs prior to leaving the test function? I certainly haven't been into the internals much but I expected that either assert would raise or the test would pass and (normally implicitly) return from the function. In either case, any local variables would die and you would have to have other references to the widgets to keep them alive properly. Is that incorrect? Or is there an assumption that the widgets are registered with qtbot?
@j9ac9k, yeah, I can see 'getting the screenshots to Azure tab X or Y' being out of scope. But somehow there would need to be a connection between a given screenshot file and a test such that it could automatically be linked in a report. I suppose probably using pytest's test identifiers, not the module/class/test-name thing I mentioned. I'm hoping that a user doesn't have to add any per-test code to get the screenshots though this really depends on my concerns above.
Is that incorrect? Or is there an assumption that the widgets are registered with qtbot?
Both assumptions are correct: all local variables are destroyed by them, and top-level widgets are supposed to be registered with qtbot.add_widget, which has the responsibility of closing them when the test ends.
We could take screenshots of the widgets under test because they are registered within qtbot, but this does not really matter because we can easily take an screenshot of the entire desktop instead using QDesktopWidget.grab.
yeah, I can see 'getting the screenshots to Azure tab X or Y' being out of scope. But somehow there would need to be a connection between a given screenshot file and a test such that it could automatically be linked in a report.
We could attach a path to a saved image file to the report object generated, like I mentioned above. The test report refers to the test by node id, which is easily traceable back to the actual test function.
I'm pretty confident we can obtain and save a screenshot of the desktop to disk when a test using qtbot fails, and provide the filename to that image on the test report. From there, we can use that filename to attach the image to a HTML report (using pytest-html) or do anything else we want (I can envision pytest-azurepipelines automatically uploading them somehow for easy viewing in the Azure UI for example).
I know this is getting really specific now, but I would think a screen shot capture method should by default only save on image on test fail, but it should optionally take an image at the end of every test.
Regarding default image location, does JUnit have a preferred location to store attachments? Some quick googling suggests there are JUnit plugins for attachments, we should probably see where they look for files for files?
I know this is getting really specific now, but I would think a screen shot capture method should by default only save on image on test fail, but it should optionally take an image at the end of every test.
I meant to take screenshots only on test failures, sorry if I wasn't clear.
Regarding default image location, does JUnit have a preferred location to store attachments?
In my experience writing non-standard junit XML files is problematic as consumers often validate the XML files against the official schema (for example Jenkins JUnit plugin), so I wouldn't pursue that route.
I think saving the image on a temporary location (same location as the tmpdir location for the test), attaching the image filename to the TestReport, and showing this along the original error message somehow, would be good starting points. What to do with the image can be decided by other third party plugins or the user using a custom plugin.
I meant to take screenshots only on test failures, sorry if I wasn't clear.
Would something like this be feasible to have an optional parameter to take a screenshot before tear down on each test? While useful to get a screenshot on failure, I can see value in the screenshot on pass as well (it can help identify things to check for within the GUI when the test should be failing).
In my experience writing non-standard junit XML files is problematic as consumers often validate the XML files against the official schema (for example Jenkins JUnit plugin), so I wouldn't pursue that route.
This is an area I have no experience/familiarity with, and will support whatever you think best.
I think saving the image on a temporary location (same location as the tmpdir location for the test), attaching the image filename to the TestReport, and showing this along the original error message somehow, would be good starting points. What to do with the image can be decided by other third party plugins or the user using a custom plugin.
That's likely the best thing to do, as you mentioned earlier, each CI system has its own method of handling attachments/artifacts.
I think this issue is a bit more broad, about saving screenshots on test failures.
@The-Compiler added the qtbot.screenshot function, which is a step in the right direction but not quite there yet.