add RNTester-E2E: tests for iOS and Android via Appium, WDIO and Jest

[kelset]

Summary:

The motivation is to create an E2E testing solution for the RNTester app that can be used to flag when things break and the release crew can use to validate more of the codebase ahead of a release.

This first PR wants to just showcase the main flow (how tests are made, where the E2E folder lives) and then we can build on top of it with subsequent PRs, with the help of an umbrella issue to get the community involved in making more tests!

This work is being done cooperation with Callstack developers (@mateuszm22 , @adzironman, and others) - reason why the branch lives in a fork.

Once this first PR that introduces the foundations is approved and merged, the next steps will be:

• leverage a device farm
• expand and add more tests (this should be handled via an umbrella issue and in coordination with the community)

Extra notes

A few things to explain about this pr:

1. why this tech stack? The reason is that it's the combination that gets the closest to the specific requirements of React Native's unique structure. We needed to fulfil these needs: A) had to be "vanilla Jest" based B) had to not touch RNTester code C) needed to support device farm D) easy to extend to out of tree platforms (ex. Windows, macOS)

2. the reason why the references to the components are iOS/Android specific is that we didn't have another option that was properly streamlined to use. We have investigated attempts at relying on accessibility labels as testIDs, but it created inconsistent scenarios nor every element on screen always had a testID available.

3. we are still on WebDriverIO 7 because during the exploratory phase, WDIO was a bit problematic and the workarounds needed to make it work don't seem very nice - see this PR: https://github.com/kelset/react-native-e2e-jest-appium-webdriverio/pull/4 --- this will be revisited in the future, once there's a better path for upgrading.

(an RFC to provide more perspective on this whole topic will be produced shortly)

Changelog:

[INTERNAL] [ADDED] - Added first working configuration for e2e testing

Test Plan:

You can play with this locally by follow the readme.

CircleCI jobs will be added.

[analysis-bot]

Platform	Engine	Arch	Size (bytes)	Diff
android	hermes	arm64-v8a	8,843,684	+241
android	hermes	armeabi-v7a	8,152,833	+235
android	hermes	x86	9,349,469	+234
android	hermes	x86_64	9,192,198	+242
android	jsc	arm64-v8a	9,456,909	+46
android	jsc	armeabi-v7a	8,638,042	+45
android	jsc	x86	9,539,996	+46
android	jsc	x86_64	9,783,297	+49

Base commit: e64756ae5bb5c0607a4d97a134620fafcb132b3b Branch: main

[facebook-github-bot]

@cortinico has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator.

[kelset]

(adding a quick side-comment here to cross-reference, we're doing something similar in RNTA https://github.com/microsoft/react-native-test-app/pull/1467/files )

[kelset]

👋 everyone.

Thank you all for your work and feedback on this PR! I did a meeting yesterday with @cortinico and @cipolleschi and we defined a new plan to ensure that this can land during H2 24; here are the key takeaways:

1. The main change is that we're now aiming to have already this first PR add a CircleCI job that runs the E2E tests on CI.

2. Aside from that, we'll also need to prepare an RFC to back this approach and give a broader vision for how we're aiming at introducing this new level of testing to improve the robustness of the codebase.

We can discuss more details offline, but that's the rough plan and these two things mentioned above are the next steps to make this land.

I've also given a quick pass at the code, rebased on top of main and some small cleanup (and bumping appium, they are entering RC phase so probably 2.0.0 is close).

[hurali97]

Hey folks,

I was reading the RFC and since there's plan to use device farm in the near future, until then I suggest implementing linkedIn's test butler to avoid silly emulator issues in CI.

However this comes with a constraint that we'll need to use stock android emulator. Now I am not aware of how it will fare with RNTester's requirement but as far as I have tested production react-native apps on CI with detox and test butler, they work without issues.

So if it's fruitful for us, we can have test butler implemented to avoid android emulator issues like ANRs, system spell checker etc.

[szymonrybczak]

👋 Hello!

Recently I added CircleCI integration, so that on every commit E2E tests are running - and I think we're ready for the final review & merge.

@NickGerleman @cipolleschi @cortinico could you please look at these changes again? 🙏

[kelset]

Can confirm on my end that everything should be ready for final review so that we can get this merged 🤩

[cipolleschi]

Left a couple of extra comments here, but great job guys! :D

[facebook-github-bot]

@cortinico has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator.

[facebook-github-bot]

@cortinico has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator.

[cortinico]

Just a small heads up, this change effectively landed internally but hasn't showed up on GitHub (that's why this PR is still opened).

At the same time, it caused some of our internal CI to fail, so it had to be reverted. I'm unsure if the bot will do anything with this PR at this point, but I'll re-import and try to land tomorrow

[facebook-github-bot]

@cortinico has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator.

[facebook-github-bot]

This pull request has been reverted by dd1c8a6ba0d4febe7e0f8da804d0e024a1cc25cc.

[facebook-github-bot]

@cortinico merged this pull request in facebook/react-native@3c196fb2f55932cec2c9be467123836e8771e249.

[github-actions[bot]]

This pull request was successfully merged by @kelset in 3c6dbec23b3267e93695b153bca0d3ebbd58fac9.

^{When will my fix make it into a release? | Upcoming Releases}

[facebook-github-bot]

@cortinico merged this pull request in facebook/react-native@3c6dbec23b3267e93695b153bca0d3ebbd58fac9.

[christian-bromann]

Hey 👋 WebdriverIO maintainer here!

It's excited to see RN using WebdriverIO for their e2e testing. Please don't hesitate to reach out to me if there are any questions or issues coming up. Looking at the code changes I was wondering why you have decided to use Jest as oppose to the WebdriverIO testrunner? The testrunner was designed to allow people to scale their e2e tests and provide good reporting and plugin capabilities. For example it could simplify setting up Appium using the @wdio/appium-service.

Any feedback would be greatly appreciated and helps making WebdriverIO better. Cheers!

[kelset]

Looking at the code changes I was wondering why you have decided to use Jest as oppose to the WebdriverIO testrunner?

They @christian-bromann - thanks for your comment! We'll surely reach out if we need help 🤗 About your question, the tl;dr is that using Jest was a strict requirement, you can read more in the RFC