[Tooling] Draft: Upload app metrics

[AliSoftware]

trafficstars

🚧 This is WIP to test the new App Size Metrics API and fastlane action implemented in https://github.com/wordpress-mobile/release-toolkit/pull/364

Internal Reference: paaHJt-3od-p2

[peril-wordpress-mobile[bot]]

You can trigger optional UI/connected tests for these changes by visiting CircleCI here.

[peril-wordpress-mobile[bot]]

You can test the changes on this Pull Request by downloading the APKs:

• Jetpack-pr-16580-build-124137.apk

• WordPress-pr-16580-build-124137.apk

[AliSoftware]

Rats: WPAndroid is still using CircleCi and its .circleci/config.yml for the Installable Builds, and there Installable Builds are built directly using gradlew assembleWordpress<Flavor><Type> -PinstallableBuildVersionName=…… — as opposed to using fastlane build_and_upload_release, fastlane build_alpha, fastlane build_beta and fastlane build_internal like I thought it would 😔

In practice CircleCI only calls those fastlane build_* lanes during the "Release Build" workflow…

So in its current state my demo here won't work on Installable Builds, only on Alpha, Beta and Final Releases.

[AliSoftware]

So in its current state my demo here won't work on Installable Builds, only on Alpha, Beta and Final Releases.

This has now been fixed:

• https://github.com/wordpress-mobile/release-toolkit/pull/365 added support for the android_send_app_size_metrics to use a "Universal APK" as the input file as well (instead of an .aab)
• This PR is now pointing to that new branch of the releae-toolkit and uses that new option to pass to the action the "Universal APK" produced by ./gradlew assembleWordPressJalapenoDebug during "Installable Build" jobs on PRs.

With that new change, you can see that the payload for both WordPress and Jetpack app metrics have been properly generated and saved as artifacts for the CI (until we get a proper runnin Metrics server to send them to).

[AliSoftware]

Note: this PR Draft was initially created to test that the new android_send_app_size_metrics from https://github.com/wordpress-mobile/release-toolkit/pull/364 + https://github.com/wordpress-mobile/release-toolkit/pull/365 worked as expected in client repos with actual APKs and AABs.

But I'm not yet sure we'd really want to send app size metrics for every Installable Build like what this Demo PR does here — and thus size data for "Universal APKs" — in practice once the Metrics server goes live (as opposed to limit it to only send data for betas and final builds, and thus size data for .aab files, but not "Universal APKs"). Cc @jkmassel thoughts?

• On one end that would make a lot of data, probably too much given how many PRs we have and that each PR can generate an Installable Build multiple times (on each new commit being pushed)
• On the other end, better have more data and filter it out, rather than less data and regretting not having it later…

[mokagio]

My two cents on whether to send app size data for every Installable Build is that we should not do it just yet. Alongside @AliSoftware legitimate questions about having too much data, there's also the fact that we are just starting out with this project. It seems wise to start small and add the extra information later on.

And while it's true that it's better to have more data and filter it out than to not have it and regret it, I think that applies more for analytics that can allow to understand user behavior (where more details give better insight) or in an even sourcing service (where you might want to replay the whole collection of events through a more advanced ML model in the future). In our case, we're talking about how the app size varies between commits on a feature branch. I think we can live with less granularity in this instance.