`verifyPaparazzi` false positives when running `testDebug` with Gradle caching enabled, becomes unreliable with Gradle Configuration Cache

[mateuszkwiecinski]

Repro:

1. Have recorded screenshots and org.gradle.caching=true enabled
2. Modify the UI
3. run ./gradlew testDebug - task succeeds
4. run ./gradlew verifyPaparazziDebug - tasks succeeds despite it should fail because the UI has been modified

looking at the details, when invoking step 4) testDebug does not run with because their inputs hasn't been modified which then ignores properties set by Paparazzi Gradle Plugin

systemProperties["paparazzi.test.record"]
systemProperties["paparazzi.test.verify"]

this also may lead to even more unexpected behaviour if one have --configuration-cache enabled - it seems like the system properties are cached, which means invoking recordPaparazziDebug before any other task will always pass paparazzi.test.record=true to test task, which in my case led to verifyPaparazzi recording new screenshots.

[jrodbx]

Possibly related to https://github.com/gradle/gradle/issues/19710?

I'm curious as to whether downgrading to 7.3.3 or 7.4.1 fixes the underlying issue.

[jrodbx]

Are you using Paparazzi 0.9.3 or latest SNAPSHOT?

[mateuszkwiecinski]

Thanks for reply 🙏

I'll describe the scenarios I'm testing right now:

1. Run ./gradlew testDebug
2. View generated report
3. Modify string visible on the UI
4. Run ./gradlew testDebug
5. View updated report
6. Revert the UI change and then either: 7a. Run ./gradlew recordPaparazzi 7b. Run ./gradlew testDebug

In scenario 7a I expect screenshots to be recorded In scenario 7b I hoped I could save some time and leverage remote build cache to avoid running tests that weren't changed. I expect to see the task either up-to-date or fetched from-cache.

I definitely observe behaviour change between Gradle 7.3.3 and 7.4. 7.4.1 - testDebug is FROM_CACHE which means there are no screenshots recorded (build scan) - it passes scenario 7b, but fails on 7a 7.3.3 - testDebug is never fetched from cache, it always gets executed. Build scan reveals candidateClassFiles and stableClasspath inputs being changed (build scan) - it passes scenario 7a, but fails on 7b so I should probably expect: is such behavior expected? Is the cacheability "disabled" by design, as a result of how Paparazzi integrates with testing framework? Seeing just the behaviour on 7.3.3 I'd report the task isn't cacheable 😛

Are you using Paparazzi 0.9.3 or latest SNAPSHOT?

I'm using the latest Snapshot. Which at this very moment is whatever being published from the main branch.

[dannyroa]

Just tried on 1.3.0-SNAPSHOT and I don't get false positives anymore.

[TWiStErRob]

Very likely fixed by #690

[jrodbx]

Fixed by #690