mobile_app_open icon indicating copy to clipboard operation
mobile_app_open copied to clipboard

Published 20 hours ago • verified publisher icon mlcommons

Documents for setup CI lab physically in the US

Open freedomtan opened this issue 2 years ago • 6 comments

  1. host machines and all the supported devices...
  2. air-conditioned?
  3. manually intervention??

freedomtan avatar Oct 25 '22 05:10 freedomtan

how "long" we are going to support older devices? we need to define it.

freedomtan avatar Oct 25 '22 05:10 freedomtan

I think we will have many same questions as in the case of building a local server vs. using a cloud server? But in the case of mobile device testing, we will have more issue to deal with since the software stack for it is not as mature as for server.

Hardware side:

  • Stable power
  • Stable internet connection
  • Secure and safe place

Software side:

  • How to deal with app crashes?
  • How to deal with OS crashes?
  • How to deal with OS update?
  • How to deal with authentication / authorization?
  • How to monitor or view test results and test logs?
  • How to queue multiple tests, cancel tests?
  • etc.

Most of the software issues may be solved with a framework for automated mobile testing. I did an internet search and found only this one http://appium.io but have no experience with it.

anhappdev avatar Oct 28 '22 03:10 anhappdev

@anhappdev to add more descriptions, such current GitHub Workflow, here.

freedomtan avatar Jul 11 '23 05:07 freedomtan

The current state of our CI/CD:

  • On every new push to this GitHub repo, multiple workflows will be triggered via GitHub Actions to run different tests.
  • One of the workflows will build the Android APK and send it to Firebase Test Lab to run integration tests on physical devices (currently Pixel 5 and 6). Firebase Test Lab will report back whether the tests passed or not.
  • An APK with all backends can also be downloaded directly from GitHub on the Actions summary page.
Screenshot 2023-07-11 at 12 57 00

It would be best if the new CI lab can be integrated into the current workflow. That means:

  1. On every new push to this GitHub repo, an APK will be sent to the CI lab
  2. The CI lab runs the tests on physical devices and report back the results (similar to how Firebase Test Lab works)

anhappdev avatar Jul 11 '23 06:07 anhappdev

Would the CI/CD results showing up in this public repo violate the terms of the CI/CD agreement with Arm, which states: "Arm will not share the results outside of MLCommons and Mobile Working Group."

nathanw-mlc avatar Jul 25 '23 16:07 nathanw-mlc

Would the CI/CD results showing up in this public repo violate the terms of the CI/CD agreement with Arm, which states: "Arm will not share the results outside of MLCommons and Mobile Working Group."

In that case, the detailed results itself can be accessed on another private website. The Arm system just reports to the GitHub workflow whether the tests run successfully or not. This is also how Firebase Test Lab works.

anhappdev avatar Jul 25 '23 23:07 anhappdev