swift-snapshot-testing
swift-snapshot-testing copied to clipboard
pointfreeco
Support snapshot testing on iOS devices
Snapshot testing is great and saves an immense amount of time vs manually checking data structures or images / tests. However, some tests need to be run on actual devices for correctness.
Why
One use case is apps that make heavy use of Metal API
- metal rendering on actual devices can vary, we want to test the result is correct on a range of them
- GPU features vary across devices, want to make sure rendering is equivalent
- Metal Performance Shaders is not supported on iOS simulator
- macOS doesn't support the texture formats as iOS, so can't run as a Mac test
- we want to test performance, but have to make sure it's still correct!
The way that swift-snapshot-testing tracks snapshot data currently does not work when the test app does not run where the source root is accessible (iOS, macOS sandbox). This has been mentioned before, eg https://github.com/pointfreeco/swift-snapshot-testing/issues/193 , perhaps https://github.com/pointfreeco/swift-snapshot-testing/issues/176#issuecomment-467328614
Proposed approach
-
__Snapshot__directories are copied into test bundle before testing begins - When test runs, snapshots are verified against snapshots on device
- If a snapshot should be updated, it is written to an
XCTAttachment - After tests execute, a post-test script copies any recorded results from
DerivedData/Project-XXXXX/Logs/Test-Project-YYYY.xcresultback to the source root
Related
- I may need to add info about the target device to paths as proposed in https://github.com/pointfreeco/swift-snapshot-testing/issues/182#issuecomment-468774418
Going to see if I can get this working, interested to hear if anyone else is trying something similar
Any luck with this? I can not run the tests on a simulator, thus I need to run them on a real device too...
You can actually make it work without modifying the library. Just use verifySnapshot which takes snapshotDirectory as an argument. Then all you have to do is place your snapshot images in a folder that gets packed into the bundle and then specify correct path for snapshotDirectory. That usually means that you can not keep your reference images in the __Snapshots__ folders next to the source, but I was ok with moving them all into one folder with some subfolders.
