iOS icon indicating copy to clipboard operation
iOS copied to clipboard

Fix Maestro reporting for e2e tests

Open loremattei opened this issue 1 year ago • 1 comments

Task/Issue URL: https://app.asana.com/0/414709148257752/1207008449808441/f Tech Design URL: CC:

Description: Maestro Cloud Action has an issue where timeouts are reported as success. At the same time, Maestro Cloud CI seems to have an issue where it gets in a weird state when test flows fail because of a timeout: every subsequent uploads/retries is stuck and ends up with a timeout. The combination of the two issues makes retries show always as a pass in GH, even if they still fail or not run at all.

Maestro CLI seems to be more reliable and reports the state of the CI correctly. Moreover, it seems to be able to correctly cancel tests that timeout.

This PR replaces Maestro Cloud Action with the CLI.

Steps to test this PR:

  1. Ensure e2e tests from this branch are green (see https://github.com/duckduckgo/iOS/actions/runs/8803029418 for a run from this branch)
  2. Validate that when a failed test is re-run, it still fails. This branch is built on top of the one in this PR and adds a commit that breaks the test. See here for consecutive runs that keep failing.

Copy Testing:

  • [ ] Use of correct apostrophes in new copy, ie rather than '

Orientation Testing:

  • [ ] Portrait
  • [ ] Landscape

Device Testing:

  • [ ] iPhone SE (1st Gen)
  • [ ] iPhone 8
  • [ ] iPhone X
  • [ ] iPhone 14 Pro
  • [ ] iPad

OS Testing:

  • [ ] iOS 14
  • [ ] iOS 15
  • [ ] iOS 16

Theme Testing:

  • [ ] Light theme
  • [ ] Dark theme

Internal references:

Software Engineering Expectations Technical Design Template

loremattei avatar Apr 11 '24 16:04 loremattei

@bwaresiak it's definitely not! But on that one we also have to fix the underlying issue with the test, so I'm planning to open a dedicated PR to address all the sync test related issues.

loremattei avatar Apr 29 '24 15:04 loremattei

This PR has been inactive for more than 7 days and will be automatically closed 7 days from now.

github-actions[bot] avatar May 07 '24 00:05 github-actions[bot]