iOS
iOS copied to clipboard
Fix Maestro reporting for e2e tests
Task/Issue URL: https://app.asana.com/0/414709148257752/1207008449808441/f Tech Design URL: CC:
Description: Maestro Cloud Action has an issue where timeouts are reported as success. At the same time, Maestro Cloud CI seems to have an issue where it gets in a weird state when test flows fail because of a timeout: every subsequent uploads/retries is stuck and ends up with a timeout. The combination of the two issues makes retries show always as a pass in GH, even if they still fail or not run at all.
Maestro CLI seems to be more reliable and reports the state of the CI correctly. Moreover, it seems to be able to correctly cancel tests that timeout.
This PR replaces Maestro Cloud Action with the CLI.
Steps to test this PR:
- Ensure e2e tests from this branch are green (see https://github.com/duckduckgo/iOS/actions/runs/8803029418 for a run from this branch)
- Validate that when a failed test is re-run, it still fails. This branch is built on top of the one in this PR and adds a commit that breaks the test. See here for consecutive runs that keep failing.
Copy Testing:
- [ ] Use of correct apostrophes in new copy, ie
’rather than'
Orientation Testing:
- [ ] Portrait
- [ ] Landscape
Device Testing:
- [ ] iPhone SE (1st Gen)
- [ ] iPhone 8
- [ ] iPhone X
- [ ] iPhone 14 Pro
- [ ] iPad
OS Testing:
- [ ] iOS 14
- [ ] iOS 15
- [ ] iOS 16
Theme Testing:
- [ ] Light theme
- [ ] Dark theme
Internal references:
@bwaresiak it's definitely not! But on that one we also have to fix the underlying issue with the test, so I'm planning to open a dedicated PR to address all the sync test related issues.
This PR has been inactive for more than 7 days and will be automatically closed 7 days from now.