react-native [RN][Release Testing] Download artifacts from CI to speed up testing

[RN][Release Testing] Download artifacts from CI to speed up testing

Open cipolleschi opened this issue 1 year ago • 3 comments

Summary:

Testing releases takes a lot of time because we have to build locally several configurations. However, the artifacts that we build locally are also built in CI.

The goal of this PR is to implement a mechanism to download those artifacts from the CI instead of build locally, so that testing the release locally can take much less time.

As an example, the full test cycle can take more than 2 hours given that we need to repackage and rebuilt the app from the template.

My plan is to add a table with the time saved once the PR is done

TODO:

[x] Download Hermes tarball for RNTester iOS
[x] Download Hermes APK for RNTester Android
[x] Download JSC APK for RNTester Android
[x] Download Packaged version of React Native to create a new app
[x] Use the downloaded React Native to initialize an app from the template
[x] Download Maven Local prebuilt in CI and use it for Template Android app

Time Savings

Setup	Before [s]	After [s]	Notes
iOS RNTester Hermes	339.68	194.86	Time saved by downloading Hermes rather then building it
iOS RNTester JSC	129.80	123.35	Not significant, expected as this workflow did not change
Android RNTester Hermes	1188.82	5.28	Huge improvement: we download the APK rather then build
Android RNTester JSC	103.10	6.28	Huge improvement: we download the APK rather then build
Creating the RNTestProject	2074.82	191.16	We download Maven, the packaged version of RN and Hermes instead of building from scratch

Changelog:

[Internal] - Speed up Release testing by downloading the CircleCI artifacts

Test Plan:

Tested the script locally

Jun 20 '23 10:06 cipolleschi

@cipolleschi has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator.

Jun 20 '23 14:06 facebook-github-bot

@cipolleschi has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator.

Jun 20 '23 14:06 facebook-github-bot

Platform Engine Arch Size (bytes) Diff

android hermes arm64-v8a 8,843,819 -1

android hermes armeabi-v7a 8,152,946 +2

android hermes x86 9,349,590 +2

android hermes x86_64 9,192,339 +3

android jsc arm64-v8a 9,455,587 +2

android jsc armeabi-v7a 8,636,726 +5

android jsc x86 9,538,676 +2

android jsc x86_64 9,781,974 +2

Platform	Engine	Arch	Size (bytes)	Diff
android	hermes	arm64-v8a	8,843,819	-1
android	hermes	armeabi-v7a	8,152,946	+2
android	hermes	x86	9,349,590	+2
android	hermes	x86_64	9,192,339	+3
android	jsc	arm64-v8a	9,455,587	+2
android	jsc	armeabi-v7a	8,636,726	+5
android	jsc	x86	9,538,676	+2
android	jsc	x86_64	9,781,974	+2

Base commit: 5705661d1f8f806a2a87336fe5b36e89d20aa80d Branch: main

Jun 20 '23 15:06 analysis-bot

/rebase

Jul 20 '23 13:07 cipolleschi

Test matrix (for testing once CI has finished post-rebasing):

local branch (`cipolleschi/download_prebuilt_ci`)

old/local flow: RNTester + iOS + Hermes: ✅ 6m 29s RNTester + iOS + JSC: ✅ 1m 41s RNTester + Android + Hermes: ✅ 7m 4s RNTester + Android + JSC: ✅ 24s RNTestProject + iOS + Hermes: ✅ (generating RNTestProject: 20m 22s) RNTestProject + iOS + JSC: ✅ (generating RNTestProject: 20m 22s) RNTestProject + Android + Hermes: ❌(generating RNTestProject: 20m 22s) RNTestProject + Android + JSC: ❌(generating RNTestProject: 20m 22s) (the Android failure is most likely unrelated to these changes but because I'm on top of main)

new/token flow: RNTester + iOS + Hermes: ✅ 2m 36s RNTester + iOS + JSC: ✅ 1m 33s RNTester + Android + Hermes: ✅ 12s 😱🤯 RNTester + Android + JSC: ✅ 8s RNTestProject + iOS + Hermes: ✅ (generating RNTestProject: 3m 34s) RNTestProject + iOS + JSC: ✅ (generating RNTestProject: 3m 34s) RNTestProject + Android + Hermes: ❌(generating RNTestProject: 3m 34s) RNTestProject + Android + JSC: ❌(generating RNTestProject: 3m 34s) (the android failure is probably for the same reason as above)

There is no significant difference between testing in this branch VS main so I'm only testing the branch. I'll delay the testing in release branches such as 71 or 72 to when we'll do the dedicated PRs to backport this there, since we'll have to do some other changes to ensure the whole thing works correctly.