spectrum
spectrum copied to clipboard
Add Visual Regression Testing capability
We'd like the user to be able to run visual regression testing. When configured:
- the first run is used to generate snapshots (golden images)
- the subsequent runs will compare the screenshots taken during each with the snapshots.
The snapshots should be taken in the src/test/resources folder by default, so to be versioned. Screenshots should be either taken:
- programmatically (user takes screenshot explicitly)
- automatically upon user's actions (clicking buttons, typing text etc)
The configuration snippet should look something like this:
visualRegression:
failFast: false # Whether to fail immediately when the first visual regression is found, rather than running the entire test
snapshots:
folder: src/test/resources/visual-regression # Where to save the screenshot references
override: false # Whether to override the snapshots references already generated
checks: # Checks configuration
count: 0 # Number of checks to perform before considering a snapshot valid
interval: .5 # Interval between checks in seconds
maxRetries: 3 # Max retries before throwing an exception
Tasks:
- [x] check multiple times before saving reference image
- [x] provide a parameter to force the snapshots update
- [x] provide a parameter to fail fast
- [ ] work with
autoBeforeandautoAfterscreenshots, relying on the EventsWebDriverListener or a dedicatedWebDriverListener - [x] inline visual regression images in html report
- [x] support
@TestFactory - [x] provide an exception with a message counting the total number of regressions
This:
work with autoBefore and autoAfter screenshots, relying on the EventsWebDriverListener or a dedicated WebDriverListener
is blocked by https://github.com/SeleniumHQ/selenium/issues/16232