playwright
playwright copied to clipboard
microsoft
[Question] CLI tool to interactively review & update snapshot test results
Hey, at one of my projects there was a custom CLI tool that allowed interactive reviewing snapshot test results and updating the baseline images/texts. As far as I know there is something similar available for snapshot tests in Jest (Interactive Snapshot Mode).
What I mean is a CLI that will go through the test results one by one and give options to open files for review (expected, actual, diff), skip, or update the baseline (copy the actual result to the snapshots).
Is something like this already available in playwright? And if there isn't - do you think it could be valuable to playwright users? If yes, then maybe I could write some proof of concept and create a PR.
@piotrgajow There is no such tool yet. You can manually go over failed screenshots in the html report, but that's it right now. We were thinking along these lines, so definitely such a tool would make sense.
If you are willing to contribute, make sure to check your idea/design/thoughts with maintainers first, to make sure your contribution will be merged.
here comes the fun and sad part about naming - PW's snapshots actually are screenshots, but jest's snapshots are HTML, so one without any hassle can locally check diff, but for screens in CI - they could be different from local ones.
GitLab (and for sure any other source control with UI ) shows the diff for pics nicely, but
- there is no heatmap ( for subtle changes )
- there is no easy way to revert/reject some screens
still, this is a task not for PW, I think but for source control, as I'm not sure how PW's report or something could interact with source code ( but I'd like to see such a thing too :) )
@dgozman How should I check the idea with maintainers first? Are there any steps to be followed in such case?
@dimkin-eu What I meant was to have a CLI for the developer to help determine if changes detected for a snapshot were intended (and the snapshot should be updated) or not. And the tool I am thinking about relies on test results that PW currently creates (i.e. actual, expected and diff images). Nothing related to repositories/pipelines.
A approve Screenshot button in the test results html view might be easier to use than a new CLI tool, because the test results are usually already open. This was my idea in #17801.
Just wondering if there's any motion regarding this issue. I like @frank-swp's idea very much.
I feel like the new recently launched UI mode would be a great place to put something like this, in Playwright 1.34.0 you can already see the screenshot comparison in the attachments tab in the UI, so having an approve button right there where you can step through the test while updating snapshots would be great.
I want this features too. Compare with BackstopJS report and feel very envious. The html report of BackstopJS is excellent although the tool itself is not very stable to run.
Just another up-vote for this feature. Would help the developer experience so much!
In the meantime, has anyone found any third-party tools that can help with the approval workflow?
Right now I am doing "open visual diff in vscode" -> "cross eyes to check the pics are identical" -> "stage changes" -> "go to the next file". But it's not the best workflow.
@neongreen unfortunately not. I tried to do some awkward integration with jest-image-snapshot (not do take snapshots, but for the comparison + UI part). But unfortunately their code is tightly coupled to the Jest runtime. I looked into their code and I don't see any chance to break this relationship that checks if Jest is used as test runner. 😕
I just wrote a script that goes through git diff and generates visual diffs using the same logic as playwright
if there’s interest I can post the script
For me, the DX is key. Developers should not get desensibilized and get annoyed by failing visual regression tests without a good way for them to verify what has changed to take a solid decision to accept a new baseline or not. Therefore it must be easy and intuitive to use without developers just starting to blindly confirm, just because they get annoyed. And it should work properly in a CI pipeline too.