mlcommons
Setup CI test for vendor-specific devices
@freedomtan Do you have a list of the devices each vendor use, so I or @d-uzlov can check if Firebase has it?
For 2.0 submission:
- Qualcomm: Xiaomi Mi 12,
- Samsung S22,
- Google: Pixel 6 Pro,
- MediaTek: Xiaomi Redmi K50 Pro
I see the following potentially relevant devices available in Firebase: Google:
- Pixel 6 (Google Tensor chipset, same as Pixel 6 Pro)
Qualcomm:
- SD888:
- SM-F926U1 (Samsung Galaxy Z Fold3)
- SD865:
- SM-G981U1 (Samsung S20)
- SM-F916U1 (Samsung Galaxy Z Fold2)
Samsung: none MediaTek: none
@jwookiehong we should consider to have each vendor (or say vendor backend) to send a device (or devices) to @d-uzlov or @anhappdev to set up CI.
Or, @anhappdev suggests, we can ask each vendor support at least one device that is in the Firebase supported device list. Then we can do functional test at least.
Another suggestion is to use another testing service if they have the desired device. A quick search gives me this: https://aws.amazon.com/device-farm/device-list/ https://saucelabs.com/platform/supported-browsers-devices https://www.perfecto.io/supported-devices#supported-devices-list
Here is full list of devices supported by firebase:
┌────────────────┬────────────────────┬───────────────────┬──────────┬─────────────┬────────────────┬───────────────┐
│ MODEL_ID │ MAKE │ MODEL_NAME │ FORM │ RESOLUTION │ OS_VERSION_IDS │ TAGS │
├────────────────┼────────────────────┼───────────────────┼──────────┼─────────────┼────────────────┼───────────────┤
│ 1610 │ Vivo │ vivo 1610 │ PHYSICAL │ 1280 x 720 │ 23 │ │
│ AOP_sprout │ HMD Global │ Nokia 9 │ PHYSICAL │ 2880 x 1440 │ 28 │ deprecated=28 │
│ ASUS_X00T_3 │ Asus │ ASUS_X00TD │ PHYSICAL │ 1080 x 2160 │ 27,28 │ │
│ F01L │ FUJITSU │ F-01L │ PHYSICAL │ 1280 x 720 │ 27 │ │
│ FRT │ HMD Global │ Nokia 1 │ PHYSICAL │ 854 x 480 │ 27 │ │
│ G8142 │ Sony │ G8142 │ PHYSICAL │ 1080 x 1920 │ 25 │ │
│ G8342 │ Sony │ G8342 │ PHYSICAL │ 1080 x 1920 │ 26 │ │
│ G8441 │ Sony │ G8441 │ PHYSICAL │ 720 x 1280 │ 26 │ │
│ H8216 │ Sony │ H8216 │ PHYSICAL │ 2160 x 1080 │ 28 │ deprecated=28 │
│ H8314 │ Sony │ H8314 │ PHYSICAL │ 1080 x 2160 │ 26 │ │
│ H9493 │ Sony │ H9493 │ PHYSICAL │ 2880 x 1440 │ 28 │ │
│ HWANE-LX1 │ Huawei │ ANE-LX1 │ PHYSICAL │ 1080 x 2280 │ 28 │ │
│ HWANE-LX2 │ Huawei │ ANE-LX2 │ PHYSICAL │ 1080 x 2280 │ 28 │ │
│ HWCOR │ Huawei │ COR-L29 │ PHYSICAL │ 1080 x 2340 │ 27 │ │
│ HWMHA │ Huawei │ MHA-L29 │ PHYSICAL │ 1920 x 1080 │ 24 │ │
│ OnePlus5T │ OnePlus │ ONEPLUS A5010 │ PHYSICAL │ 1080 x 2160 │ 28 │ │
│ SC-02K │ Samsung │ SC-02K │ PHYSICAL │ 2220 x 1080 │ 28 │ │
│ SH-01L │ SHARP │ SH-01L │ PHYSICAL │ 2160 x 1080 │ 28 │ │
│ SH-03K │ SHARP │ SH-03K │ PHYSICAL │ 3040 x 1440 │ 28 │ │
│ TC77 │ Zebra Technologies │ TC77 │ PHYSICAL │ 1280 x 720 │ 27 │ │
│ a10 │ Samsung │ SM-A105FN │ PHYSICAL │ 720 x 1520 │ 29 │ │
│ b2q │ Samsung │ SM-F711U1 │ PHYSICAL │ 260 x 512 │ 30 │ │
│ blueline │ Google │ Pixel 3 │ PHYSICAL │ 2160 x 1080 │ 28 │ │
│ cactus │ Xiaomi │ Redmi 6A │ PHYSICAL │ 1440 x 720 │ 27 │ │
│ cruiserlteatt │ Samsung │ SM-G892A │ PHYSICAL │ 1080 x 2220 │ 26 │ │
│ dreamlte │ Samsung │ SM-G950F │ PHYSICAL │ 1080 x 2220 │ 28 │ │
│ f2q │ Samsung │ SM-F916U1 │ PHYSICAL │ 2208 x 1768 │ 30 │ │
│ flo │ Asus │ Nexus 7 │ PHYSICAL │ 1200 x 1920 │ 19 │ │
│ grandppltedx │ Samsung │ SM-G532G │ PHYSICAL │ 540 x 960 │ 23 │ │
│ griffin │ Motorola │ XT1650 │ PHYSICAL │ 1440 x 2560 │ 24 │ │
│ gts3lltevzw │ Samsung │ SM-T827V │ PHYSICAL │ 1536 x 2048 │ 28 │ │
│ gts4lltevzw │ Samsung │ SM-T837V │ PHYSICAL │ 2560 x 1600 │ 28 │ │
│ gts4lvwifi │ Samsung │ SM-T720 │ PHYSICAL │ 1600 x 2560 │ 28 │ │
│ hammerhead │ LG │ Nexus 5 │ PHYSICAL │ 1920 x 1080 │ 23 │ │
│ harpia │ Motorola │ Moto G Play │ PHYSICAL │ 1280 x 720 │ 23 │ │
│ heroqlteaio │ Samsung │ SAMSUNG-SM-G930AZ │ PHYSICAL │ 1080 x 1920 │ 26 │ │
│ htc_pmeuhl │ HTC │ HTC 10 │ PHYSICAL │ 1440 x 2560 │ 26 │ │
│ hwALE-H │ Huawei │ ALE-L23 │ PHYSICAL │ 1280 x 720 │ 21 │ │
│ j7popltevzw │ Samsung │ SM-J727V │ PHYSICAL │ 1280 x 720 │ 27 │ │
│ joan │ LG │ LG-H932 │ PHYSICAL │ 1440 x 2880 │ 26 │ │
│ judypn │ LG │ LM-V405 │ PHYSICAL │ 3120 x 1440 │ 28 │ │
│ lt02wifi │ Samsung │ SM-T210 │ PHYSICAL │ 600 x 1024 │ 19 │ │
│ lv0 │ LG │ LG-AS110 │ PHYSICAL │ 854 x 480 │ 23 │ │
│ oriole │ Google │ Pixel 6 │ PHYSICAL │ 2400 x 1080 │ 31 │ │
│ pettyl │ Motorola │ moto e5 play │ PHYSICAL │ 960 x 480 │ 27 │ │
│ phoenix_sprout │ LG │ LM-Q910 │ PHYSICAL │ 3120 x 1440 │ 28 │ │
│ q2q │ Samsung │ SM-F926U1 │ PHYSICAL │ 2208 x 1768 │ 30 │ │
│ redfin │ Google │ Pixel 5e │ PHYSICAL │ 2340 x 1080 │ 30 │ default │
│ sailfish │ Google │ Pixel │ PHYSICAL │ 640 x 480 │ 25 │ │
│ starqlteue │ Samsung │ SM-G960U1 │ PHYSICAL │ 1080 x 2220 │ 26 │ │
│ walleye │ Google │ Pixel 2 │ PHYSICAL │ 1920 x 1080 │ 27 │ │
│ x1q │ Samsung │ SM-G981U1 │ PHYSICAL │ 3200 x 1440 │ 29 │ │
└────────────────┴────────────────────┴───────────────────┴──────────┴─────────────┴────────────────┴───────────────┘
Obtained via gcloud firebase test android models list --filter=physical.
It seems most of them are Google devices (no surprise). And few of them are recent devices (see the OS_VERSION_IDS column, which is supposed to be Android API level)
Working group has concluded to send the physical devices to Anh . @anhappdev we can discuss this in next week meeting to confirm.
@d-uzlov suggested we can use some available devices in Firebase, e.g., Pixel devices, at first. And testing accuracy seems to be good. At least if a backend is running or not.
Another suggestion is to use another testing service if they have the desired device. A quick search gives me this: https://aws.amazon.com/device-farm/device-list/ https://saucelabs.com/platform/supported-browsers-devices
Let's explore this kind of alternative device service providers.
We don't find any reasonable (cost and time) solution to run CI test on latest gen physical devices so we close it for now.
@swasson488 Let's reopen this. To to have CI/CD running on service providers who are already running device farm.
