donv
Unexpected file diffs when fail_on_difference is true
When fail_on_difference is set to true, running the specs will pass but screenshots files will be changed (I see them as modified with git status), with no visible diffs in the heatmap
If I set it to false, the screenshots will not be changed.
This didn't happen before, I think it is now happening since I changed my screen (from 1080p to 1444p), could it explain this behavior ?
@Intrepidd yep, the problem is that when you change the screenshot size, you need to re-generate new screenshots with the new sizes first. We have not added auto-rescaling yet.
The screenshot size did not change (in the capybara config), just the screen resolution. The produced screenshots are still the same size
Oh, I got it. Sorry for the confusion. fail_on_difference has been added to regenerate all screenshots without failing tests: https://github.com/snap-diff/snap_diff-capybara/pull/109 to ensure that they are up to date.
If I set it to false, the screenshots will not be changed.
We do not store new versions; we just reuse previous screenshots. When your tests have been passed because of some enabled filters, you do not need to have a diff report for them.
My understanding is that it should not overwrite screenshots if there is no diff, which I believe (but I can be wrong) was the case before for me. Do I make sense ?
@Intrepidd, comparing screenshots is expensive, so we take screenshots and override the latest version by default. If we run verification, we will rollback the action on the difference, but if we do not find a difference, we will not rollback.
gotcha, thanks for the precision !
Would you be interested in a way to configure the gem so it matches the behaviour I was expecting :
- Don't fail in case of mismatch
- But don't take screenshot / rollback screenshot if it matches
gotcha, thanks for the precision !
Would you be interested in a way to configure the gem so it matches the behaviour I was expecting :
- Don't fail in case of mismatch
- But don't take screenshot / rollback screenshot if it matches
@donv what do you think?