testbench icon indicating copy to clipboard operation
testbench copied to clipboard

Change default Parameters for screenshot testing to be more relaxed

Open TatuLund opened this issue 1 year ago • 0 comments

The default Parameter's in screen shot testing are strict. This means that tests will be flaky.

For example I am using in my add-on testing the following settings

        // We do screenshot testing, adjust settings to ensure less flakiness
        Parameters.setScreenshotComparisonTolerance(0.05);
        Parameters.setScreenshotComparisonCursorDetection(true);
        testBench().resizeViewPortTo(800, 600);
        Parameters.setMaxScreenshotRetries(3);
        Parameters.setScreenshotRetryDelay(1000);

The default view port of app testing should be larger than 800 x 600 I use for add-ons. Point here is that we should ensure some fixed default size, since otherwise the view port size can wary.

More over I sometimes run tests also in Development mode, when dev mode gizmo needs to be hidden

        // Hide dev mode gizmo, it would interfere screenshot tests
        try {
            $("vaadin-dev-tools").first().setProperty("hidden", true);
        } catch (NotFoundException e) {

        }

Screenshot testing would be more approachable if the defaults would be something like this. Now developers need to consume time on trial and error to find these.

TatuLund avatar Oct 02 '23 06:10 TatuLund