Ability to determine if compare methods created a new baseline, or used an existing one.

[ablok]

Is your feature request related to a problem? Please describe. When using the compare methos, I wrap them in a WDIO waintUntil method. This helps with problems regarding synchronisation caused by (unpredictably) slow environments. It also keeps the code cleaner since you do not have to wait for a specific element to appear before taking a screenshots. You just keep making screenshots until there is a match, or until the timeout hits. This works fine.

But when creating a baseline, you might get the wrong screenshot (too early) if you do not wait for the correct element. as described above. However, I assume the application under test should reach a static state at some point. If I know when a new baseline image is created I can wait until that baseline is the same for a certain amount of time (for instance 5 retries with 1 sec interval). This way I know my application reached a steady state and this is the actual baseline image I want to capture.

However, currently it is not possible to determine if a new baseline was created from compare commands output.

Describe the solution you'd like To be able to determine if a compare method created a new baseline or used and existing one. The checkBaselineImageExists method could return a boolean value if the baseline existed or not. If it did not exist, it will either throw an error if autosaveBaseline was not set to true, or will create a new baseline if autosaveBaselin was set to true. In the last case, you can skip the determineIgnoreRectangles and executeCompare method because you have nothing to compare and the diff wil always be 0. Instead, you could make the misMatchPercentage and the diff folder optional and leave them out of the return.

Describe alternatives you've considered Also make checkBaselineImageExists return a false value when a new baseline is created. You keep the rest the same and add an additional property to the output of the compare methods. Something like a boolean that's called baselineCreated which is the inverse of the output of checkBaselineImageExists.

Additional context

[ablok]

While writing this down, maybe the alternative sounds better. It will change the API though, but it's only an addition. I could implement this if you think this is a useful addition.

[ablok]

As an example, you can do this:

import { rmSync } from "fs";
import { join } from "path";

export async function expectToMatchImageSnapshot(context: Mocha.Context, element?: WebdriverIO.Element) {
    const result = await performSnapshot(context, element);
    if (result.baselineCreated === false) {
        // Existing baseline, keep checking until snapshot matches baseline or timeout is reached
        await driver.waitUntil(
            async () => {
                try {
                    expect((await performSnapshot(context, element)).misMatchPercentage).toEqual(0);
                    return true;
                } catch (error) {
                    console.log((error as Error).message);
                    return false;
                }
            },
            { timeoutMsg: `No image match before timeout!` }
        );
    } else {
        // New baseline, keep checking until AUT is static for multiple intervals
        let count = 0;
        await driver.waitUntil(
            async () => {
                const result = await performSnapshot(context, element);
                if (result.misMatchPercentage === 0) {
                    count++;
                }
                if (count === 5) {
                    return true;
                } else {
                    rmSync(join(result.folders.actual, result.fileName));
                    count = 0;
                    return false;
                }
            },
            { interval: 1000 }
        );
    }
}

async function performSnapshot(context: Mocha.Context, element?: WebdriverIO.Element) {
    let result;
    if (element) {
        result = await browser.compareElement(element, context.test?.fullTitle()!);
    } else {
        result = await browser.compareScreen(context.test?.fullTitle()!);
    }
    return result;
}

[wswebcreation]

Wow, nice @ablok

I've requested a small change on the PR. IS there maybe a way that you can add this also as a custom matcher? For and Jasmine/Mocha/Cucumber. That can be done in a separate PR if you like

[ablok]

That's a nice idea. I will see what I can do there. New to me, so might take some time. I'd suggest not waiting for that.

[ablok]

Regarding this matcher. Should it

1. extend Jests` expect
2. extend expect of expect-webdriverio?

Because expect-webdriverio is not necessarily installed. What do we do if we go for option 1 and it is not installed in the users project? If we go for option 2 and expect-webdriverio IS installed and globally available you will need a different import for jest to be able to use this matcher.

[ablok]

See #44