Document how to make screenshot snapshot comparison keyword

[mkorpela]

Is your feature request related to a problem? Please describe. Visual comparison is pretty much used technique in UI testing. This can be done manually, but in bigger cases it is better to at least precheck images automatically. Snapshot testing in general is used to detect regressions. Snapshot tests can break for many reasons as they will in any case detect changes.

Describe the solution you'd like Browser library should offer an easy way to compare selector specified part of a web page to a predefined snapshot. The same mechanism should be easily configured to replace OR update the snapshot.

Describe alternatives you've considered There are some existing libraries for image comparison with RF.

[simonmeggle]

@mkorpela First I thought you are planning to rewrite ImageHorizon library :-) But this approach sounds interesting. How are you planning to get the view of the page? Will this be a headless rendering of the page?

[manykarim]

So - what could be a good way of enabling automated visual comparisons for Browser Library? I would like to implement some solution, where the visual comparisons are happening in the "background" without the need to compare each element individually. Maybe a possiblity to set some filter (button, input, label, div, ..) to tell the extension, which elements shall be compared visually.

I'm wondering: How to automatically keep track of the elements and their reference screenshots? If the test steps are stable, there could be a way of having a unique ID for each step. But what if additional steps are introduced..?

[Snooz82]

@manykarim I think the reference should be done manually or it should be done by browser. So there could be a keyword Create Reference Image .myclass > button which then saves a screenshot under a specific directory under a filename created from the selector string.

Another keyword could automatically compare a selected element with the reference image found by the selector or by a given name: Verify Element Visually .myclass > button ref.myclass.png or Verify Element Visually .myclass > button

i think the latter keyword could be done by DocTestLib when @aaltat has implemented the python plugin API.

[Snooz82]

@manykarim so basically what you have done here:

        ${reference_screenshot_exists}    Run Keyword And Return Status    File Should Exist
        ...    ${CURDIR}/reference/${pagename}_${nodeType}_${nodeName}_${id}_${className}.png
        IF    ${reference_screenshot_exists}
            Take Screenshot    filename=${CURDIR}/candidate/${pagename}_${nodeType}_${nodeName}_${id}_${className}
            ...    selector=${element}
            Run Keyword And Ignore Error    Compare Images
            ...    ${CURDIR}/reference/${pagename}_${nodeType}_${nodeName}_${id}_${className}.png
            ...    ${CURDIR}/candidate/${pagename}_${nodeType}_${nodeName}_${id}_${className}.png    move_tolerance=1
        ELSE
            Take Screenshot    filename=${CURDIR}/reference/${pagename}_${nodeType}_${nodeName}_${id}_${className}
            ...    selector=${element}
        END

[manykarim]

So you'd prefer an explicit Keyword to do the visual comparison? I was also thinking of an automated visual comparison in the background, like some commercial tools do. It would allow people to just switch visual comparison on or off. But there's also a risk that a lot of things are compared, that you don't really like to be compared...

[Snooz82]

I think automated visual comparison is mostly done during Cross Browser testing.

you run a test with your reference browser and store the pages screenshot. Then you run the same test against the same test object with firefox and webkit and then you compare the images and log the differences.

for normal test runs against different versions of the test object i would assume that most of the differences are intended and therefore it would cause a lot of false positives.

[mkorpela]

for normal test runs against different versions of the test object i would assume that most of the differences are intended and therefore it would cause a lot of false positives.

I think this would be the normal use - compare to previous. Masking and other tricks can be used to remove some of the false failures. But yep snapshot testing will also spot expected changes.

[Snooz82]

@manykarim any ideas for a Browser lib plugin done by DocTestLib?

Otherwise i would close this issue?!

[Snooz82]

documented here: MarketSquare/robotframework-browser-extensions/Compare_Images