public-roadmap
public-roadmap copied to clipboard
checkly
Support lightweight web change detection with snapshot match
💡 For general support requests and bug reports, please go to checklyhq.com/support
Is your feature request related to a problem? Please describe. playwright is nice but still a lot of work if all you want to make sure your website is live. instead, it would be much easier to take a snapshot/screenshot (typically steady-state, not changing website) and code the check to compare it, like https://playwright.dev/docs/test-snapshots
Describe the solution you'd like above.
Describe alternatives you've considered perhaps a no-code option just for a screen diff. doesn't have to use playwright
Additional context toMatchSnapshot? The issue is that it's part of @playwright/test which Checkly doesn't support at the moment. You can see however that it isn't completely out of our radar https://github.com/checkly/public-roadmap/issues/167.
@felixsafegraph thanks for reporting. This is a very interesting feature, regretfully with many many corner cases in the current Checkly model. We do want to support playwright test ASAP, but checking assets over multiple runs from (potentially) different locations around the globe will create some interesting "bugs" where pages look slightly different. Think of localized sites, timestamps etc. that will be different.
Can you let me know how you deal with those situations right now? Do you have a solution that already works for you?
we don't have a solution. I do think for very simple sites. a visual match is the quickest way we could get a lot of coverage quickly. I often find it non-trivial and even time consuming trying to build the test for very simple validation.
Playwright Test uses the pixelmatch library. You can pass comparison threshold as an option. https://playwright.dev/docs/test-snapshots
@felixsafegraph would you or your team do this "match" on deploy, or around the clock on a schedule?
think both are important. first, having an easy way to periodically check for the health (deployment or not) would get us quickly to value of the monitoring second, at deployment, this could be a quick smoke test to validate the change/state of the website
I would also like to echo the need for this feature. Essentially what I would love is to have on any given run the ability to set a screenshot as the "baseline" then be able to make assertions that a new Screen Shot passes the comparison threshold as mentioned by @felixsafegraph in the comment above.
When a new check fails this assertion there should be an option to accept the new screenshot as the new baseline for when those changes are expected. Checkly is already an amazing platform from which I gain a ton of value. This would make it even better. It would also allow me to get my QA department on board because it's much more in keeping with the way they view testing. I'm trying to work on that and get them behind low level assertions with playwright but there's always pushback because it feels like asking QA analysts to write software which is not always in keeping with what they want for themselves.
An interface that allows you to basically say:
This is what the page should look like let me know if it ever starts looking like anything else
would be insanely valuable and could help me sell many many more check runs than I am already using to my company which would increase your bottom line. Especially since you already have such a solid ecosystem around creating play write scripts and there is already a library that supports this in that same framework feels very much like a no-brainer to me FWIW ¯\_(ツ)_/¯.
@felixsafegraph @infn8 Hi there, just heads up that we've started working on the first iteration of Playwright Test support at Checkly.
While that's not directly addressing visual comparisons as suggested above, do you see some value in trying this out with PW Test? If so, how do you envision the system working?
If you'd like to jump on a call or prefer email, I'm at darko at checklyhq . com. Thanks!
