[Bug]: Incorrect screenshot tests failure detection on CI

[SimonMarquis]

Is there an existing issue for this?

• [X] I have searched the existing issues

Is there a StackOverflow question about this issue?

• [X] I have searched StackOverflow

What happened?

The current implementation for detecting screenshot changes is based on checking if the verifyRoborazziDemoDebug task fails.

• https://github.com/android/nowinandroid/blob/335a7ec68c76c618fad19cbe2b368ac841c62ca0/.github/workflows/Build.yaml#L45-L48

• https://github.com/android/nowinandroid/blob/335a7ec68c76c618fad19cbe2b368ac841c62ca0/.github/workflows/Build.yaml#L58-L62

Unfortunately, executing this task can fail in multiple ways, not always related to actual screenshot changes. For example, this PR introduces a compilation error in the test sources, but is still detected as a screenshot test and tries to re-update the golden images.

It could be great to be able to distinguish errors coming from unexpected screenshot changes, and all others.

Some solutions could be:

• Run the compilation tasks for roborazzi before executing verifyRoborazziDemoDebug. It seems like Roborazzi does not yet offer such lifecycle tasks, but could probably easily be implemented on top of JVM's testClasses or Android's compile<Variant>UnitTestSources.
• Detect the presence of outputs/roborazzi/*_compare.png diff images. This one would be trivial to implement.

Relevant logcat output

No response

Code of Conduct

• [X] I agree to follow this project's Code of Conduct

[takahirom]

Hi One issue is that if the composable function throws an exception, it fails. It's also failing tests, although the screenshot test remains unchanged. As you mentioned, in addition to the _compare.png condition, Roborazzi also provides a JSON report. However, I believe a simpler solution, such as using a running task or checking _compare.png, would be preferable to striving for perfection.

Please let me know if there's anything I can assist with on the Roborazzi side.