infra(e2e): rework local E2E script (WIP)

[kelset]

Summary

This is a work in progress - I'm opening the draft PR so that the rest of the release crew can play with it and give feedback while I'm off

This is a long time coming effort to improve the situation around the local e2e script that in the release crew: the current bash-based script is quirky at best, and what you end up generating as a sample project is not really a true sample project. This is where this PR comes in: it migrates the flow from ./scripts/test-manual-e2e.sh to yarn test-e2e-local <options>.

Here's the current shape of the options:

Options:
  --help          Show help                                            [boolean]
  --version       Show version number                                  [boolean]
  -t, --target      [choices: "RNTester", "RNTestProject"] [default: "RNTester"]
  -p, --platform                    [choices: "iOS", "Android"] [default: "iOS"]
  -h, --hermes                                         [boolean] [default: true]

The idea is to change it so that you can just run the script, and it will do that one specific thing "well", without the tester needing to do anything aside from actually testing the app once it's open.

Some of the key changes:

• tries to stick to the patterns of the other established *.js based scripts, in terms of tooling and approach (and even refactor parts that can be shared with other scripts) - like the android artifacts generation
• no need to start the android emulator on the side
• no need to start Metro on the side
• RNTester iOS will open up on the simulator (no Xcode open that then you need to press)

Things that still need work:

• see the #fixme and #todo in comments
• clean up strategy/a deep clean command
• fix the issue with RNTestProject iOS Hermes mentioned below
• clean up the code
• because we rely on exec, the output sent back is not formatted/shaped correctly so it's a bit more noisy/chaotic
• DX for the RNTestProject can be worked on a bit more and polished

Changelog

[Internal] [Changed] - Migrate bash E2E local testing script to new JS based command

Test Plan

To test the script, you can run it passing the options showed above; this is the current situation:

• RNTester iOS Hermes ✅
• RNTester Android Hermes ✅
• RNTester iOS JSC ✅
• RNTester Android JSC ✅
• RNTestProject Android Hermes ✅
• RNTestProject iOS Hermes ❌ need to be worked on, the issue is probably related that now we generate a proper "fully shaped" node_module and so it tries to download the hermes artifacts from the GH release for a release that doesn't exist 😅

[github-actions[bot]]

	Warnings
:warning:	:lock: package.json - Changes were made to package.json. This will require a manual import by a Facebook employee.

Generated by :no_entry_sign: dangerJS against bbd4d2dd4f1fbdbafa852b38aea2a89387370cbe

[analysis-bot]

Platform	Engine	Arch	Size (bytes)	Diff
ios	-	universal	n/a	--

Base commit: cc13b0273fc533a38c06fddd5e6b74f77319afa6 Branch: main

[analysis-bot]

Platform	Engine	Arch	Size (bytes)	Diff
android	hermes	arm64-v8a	7,739,292	-70
android	hermes	armeabi-v7a	7,141,935	-93
android	hermes	x86	8,050,381	-91
android	hermes	x86_64	8,022,197	-104
android	jsc	arm64-v8a	9,607,479	-70
android	jsc	armeabi-v7a	8,372,968	-103
android	jsc	x86	9,555,141	-77
android	jsc	x86_64	10,147,725	-96

Base commit: 97f5ef05e6396e9694e4b021e830b71c5900cd3c Branch: main

[kelset]

update on solving the Hermes on iOS for RNTestProject: I managed to figure out the logic to make it build from source on the new project for when testing in main branch and from release branch.

BUT this caused this to happen:

Basically, on the Android side everything gets build with the correct configuration, while on iOS not. This took a lot of digging, and a bit of fortune and "lucky timing", but basically now we know what's going on (it's the same issue @Kudo is trying to address here). Long story short, how hermes gets build on iOS has 3 different variations:

• there's build-apple-framework.sh in the Hermes repo, that runs this Cmake command
• there's build-apple-framework.sh in the React Native repo, that runs this variation of the Cmake command that in particular has this extra option -DHERMES_RELEASE_VERSION="for RN $(get_release_version)" \ that makes the right version of Hermes be generated (and displayed in a fresh new project)
• there's a variation of the above that is run in the RN CI, but it's all "manual" while the ones above are run through cocoapods. See here the scripts.

The core problem is that, within the hermes-engine.podspec contained in the RN repo (at react-native/main/sdks/hermes-engine/), there's a bit of logic that triggers that script here (through ./utils/build-ios-framework.sh). The issue is that we all thought that that  ./utils/build-ios-framework.sh would invoke the React native version of the scripts (since the podspec file lives right next to the utils folder) but, in reality, it doesn't. It just so happens that the Hermes repo has a root level utils folder which is (you guessed it) where the Hermes variation of those build scripts live. So, when running the pod install command in a react-native project (build from source), it will go and download the hermes source code (since the source[:git] gets set) but then it will use the hermes variation of the build-*.sh scripts.

So we need to figure out how to get the pod process to build using the React Native variation of those scripts. @Kudo has proposed a patch here that would have some more explicit pathing - which seems a good workaround, but it doesn't seem to do the trick:

So currently I'm freezing working on this until this build hermes issue is addressed.

[kelset]

update, with the latest updated patch proposed by Kudo, this works correctly now 🎉

---

the fix now is live here https://github.com/facebook/react-native/pull/34710, once that's merged I'll rebase this on top - and everything should be taken care of.

[facebook-github-bot]

@dmytrorykun has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator.

[kelset]

@dmytrorykun just an headsup, I'm testing on a release branch and I've hit an issue... I'll let you know once I've fixed it

[kelset]

@dmytrorykun / @cortinico ok nvm the issue I was hitting was caused by me not having cherry-picked all the necessary changes back on top of 0.70 to test my script correctly; after a bit of investigation it's clear that it's tooo complicated to backport this script so we'll have to face potential/eventual issues during the 0.71 RC phase - but on paper everything should work fine 🤞😇

[kelset]

(note to self: this page of the docs https://reactnative.dev/contributing/how-to-contribute-code#development-workflow needs to be updated once this is merged)

[cortinico]

Can we solve the merge conflict?

[kelset]

@cortinico done 👍

[cortinico]

Are we fine merging this?

[kelset]

from my perspective we should be good to go; we won't cherry pick it back into 0.70 or older but at least starting from 0.71 we'll have a better DX for testing, and we can tell people to use it too.

It also brings along a couple of small improvements so it'd be good to merge :3

Maybe you want to checkout the branch locally and run it once just to verify it works on other setups other than my machine?

[facebook-github-bot]

@dmytrorykun has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator.

[react-native-bot]

This pull request was successfully merged by @kelset in 97f5ef05e6396e9694e4b021e830b71c5900cd3c.

^{When will my fix make it into a release? | Upcoming Releases}