Introduce a Diff Report when Verifying Screenshots

[xiphirx]

Given that we encourage running tests on CI, it would be nice to also include a report that shows what screenshot changed similar to the report generated at the end of a local run.

[dimorinny]

What do you think about generating Junit report for images checks?

[josedlpozo]

Hello @xiphirx, I have been working in this feature for some days.

I wanted to have a report with each test run and not stop in the first screenshot test failed. So, I changed the record class behavior to not raise an Exception if the screenshot is different. This way we can generate a report file with the test results.

I have attached some screenshots with the report html file generated after running some fail and success tests. Also the tests results is printed in the command line.

This feature(https://github.com/josedlpozo/screenshot-tests-for-android/tree/features/image_diff) in my fork depends on this PR(https://github.com/facebook/screenshot-tests-for-android/pull/115) already open 😄

Please, tell me if you want this to be merged or some way we can improve this feature 🤔

Thank u!

[xiphirx]

@josedlpozo that looks great! The diff report would need to include the two existing actions the non-diff report contains: toggling a dark background and having a view hierarchy overlay. There would also need to be a way to have a full sized screenshot shown, either via a button or just by hovering over the screenshot with your mouse.

@dimorinny Sure, a junit xml report in addition to the html diff report would be useful I'd imagine

[ubuntudroid]

Having something like that would be awesome!

@josedlpozo Now that the multi-device PR has been merged, will you submit a PR for the diff report as well?

[josedlpozo]

Hey @ubuntudroid! I'm working on it but these days are being very busy! I will submit a PR in the following days 😄

[pedrovgs]

@josedlpozo @xiphirx @ubuntudroid we've been working on this feature in our own Gradle plugin using the jar artifact you developed as a core dependency 😃We wanted to implement this plugin in Scala and that's why we created a new Gradle plugin. When we started this project this repository seemed to be inactive for months, but know we see is still under development. From Karumi we'd like to know if we could help or if there is some way we could collaborate. You can find our Gradle plugin here: https://github.com/Karumi/Shot/ It's fully compatible with the original one and can be combined in the same build if needed.

As you can see in the following screenshots we've improved the console and HTML report:

If you think we can help or collaborate just let us know 😃

[xiphirx]

I'm currently in the process of rewriting the plugin portion in Kotlin. My goals are to make it play much more nicely with build variants and such. As for this diff report, I want to spend some time rethinking how we're doing things. I'm probably going to lean towards removing python from the repo, but nothing is set in stone.

Unfortunately I'm very busy at work with other things, so this is sort of a side project at the moment.

[pedrovgs]

Happy to hear that @xiphirx If instead of Kotlin you'd like to use Scala let me know. I'd really like to collaborate on this project and we could use most of the code already written in Shot.

[xiphirx]

I don't think we would use Scala unfortunately, as its not an officially supported language for Android.

I think the primary goal for the diff report would be the ability to retrieve the before image from some server / endpoint. This would allow you to see changes from the most up to date version of the screenshot to your local version.

The project itself is not really dead at all, there are a lot of things currently in flux regarding this project that I'm trying to get a handle of before I can dedicate more time to it. However, it is still a core part of testing we do at FB.

[dmarjanovic]

Is there maybe any plans to make a progress on this topic? It's a year old and there are no recent updates. We would be really interested into using the feature if we can automate the diff report in our CI. Currently it's not user friendly to check diff manually by comparing images which might be big limitation as being time consuming in daily work.

[xiphirx]

Yes, I have plans to work on a lot of things relating to this project. I've just been really busy with other things.

Specifically, I plan on transforming the core library to be rule based, transitioning the plugin to be based on Kotlin/Java rather than Python and then working on the local development experience.

I'm also happy to review contributions to this area. At this point the diff report would need to be done within the python scripts.

For a first version we could use a CSS3 image filter trick to highlight differences (https://franklinta.com/2014/11/30/image-diffing-using-css/)

[KaurManjot]

Is there still any plans to make a progress this? Would be another year.

[FlorianMarsch]

Is there still any plans to make a progress this? Would be another year.

[tdrhq]

Hi all,

I built a thing: https://screenshotbot.io (EDIT: If you're reading this in 2022, modern documentation here: https://screenshotbot.io/documentation/platforms/android-apps)

Instead of the record/verify that you might be used to, this generates screenshots in the cloud. And if anything changes, it opens a GitHub task, like so: https://github.com/tdrhq/screenshotbot-example/issues/8

The idea is that most screenshot changes are intentional, not bugs, they just need to be reviewed. This is work in progress, but it's definitely already very usable. Here's an example of generating the screenshots with CircleCI and Firebase: https://github.com/tdrhq/screenshotbot-example/blob/master/.circleci/config.yml. Jenkins without Firebase should be a lot more easier, since I have a gradle plugin to handle most of the hard work.

Feel free to ping me if you have any questions, here's an invite to a slack channel if you'd like to chat with me about this: https://join.slack.com/t/screenshotbot/shared_invite/zt-i774e3n7-ItDQ76NMUMh~V2Qe8y42_w