qutebrowser
Screenshots for failed end2end tests
It could be useful to get screenshots of what was displayed in the browser under test in CI. Frontend testing frameworks like nightwatch and cypress has this feature and it can be useful for getting an extra view on what went wrong.
A lot of our end2end test failures are a step separated from the actual cause of the error (eg "timed out waiting for element") and for things like failed navigations having a picture to show it could be interesting.
There are a couple of additional complications beyond "take a screenshot":
- it'll probably have to be integrated with pytest in some way, we use https://github.com/The-Compiler/pytest-xvfb/ to tie the virtual display manager into the test lifecycle and that uses https://github.com/ponty/PyVirtualDisplay to run it (pyvirtualdisplay can already take screenshots!)
- you'll have to put the screenshots somewhere, ideally it would be on the same page as the github actions run logs (for example). You can add "artifacts" to an actions run which might be good enough, but that bundles everything up into a zip file, ideally it would be one or zero clicks to get to the images
It could be useful to get screenshots of what was displayed in the browser under test in CI. Frontend testing frameworks like nightwatch and cypress has this feature and it can be useful for getting an extra view on what went wrong.
Agreed! Some months ago I added basic screenshot functionality to pytest-qt:
- https://github.com/pytest-dev/pytest-qt/pull/433
But that won't help us much for the end2end tests, and it doesn't have anything to automatically store screenshots on test failures either.
- it'll probably have to be integrated with pytest in some way, we use https://github.com/The-Compiler/pytest-xvfb/ to tie the virtual display manager into the test lifecycle
Hey, I know the maintainer of that! :upside_down_face: Would be a cool feature to have in there, bet that would benefit some other projects too.
and that uses https://github.com/ponty/PyVirtualDisplay to run it (pyvirtualdisplay can already take screenshots!)
Looks like it uses Pillow's ImageGrab to do that.
Another possibility if we wanted to do this in qutebrowser's code or in pytest-qt would be to ask Qt to take a screenshot. But that'll probably only screenshot a single window.
- you'll have to put the screenshots somewhere, ideally it would be on the same page as the github actions run logs (for example). You can add "artifacts" to an actions run which might be good enough, but that bundles everything up into a zip file, ideally it would be one or zero clicks to get to the images
There are GitHub Actions Job Summaries which would be pretty nice for this, perhaps with something like pytest-md: https://github.com/hackebrot/pytest-md/issues/21.
Though I'm not sure if/how we could add images for this. Maybe data URLs inside the markdown would work, even if ugly. Otherwise I suppose we would find a way to automatically upload/host the images somewhere...
Ah, it seems this has been requested before, a long time ago: https://github.com/qutebrowser/qutebrowser/issues/2269
Ah, "screencaps"... that explains why I vaguely remembered there was an issue for this already and couldn't find it!
Closing the other one, as this one has a bit more context.