trezor-firmware icon indicating copy to clipboard operation
trezor-firmware copied to clipboard

Model R UI tests

Open grdddj opened this issue 3 years ago • 6 comments

Connected with https://github.com/trezor/trezor-firmware/issues/2246:

  • making sure the UI tests can be run for model R

A lot of layouts had to be mocked just to be able to run all the tests, so they show at least something on the screen and do not raise NotImplementedError - see core/src/trezor/ui/layouts/tr/__init__.py.

Local results of the whole device_test suite is 258 failed, 736 passed, 90 skipped, 7 xfailed, 6 errors.

Next step is to go test-by-test and try to make it work while improving the visual side of it.

grdddj avatar May 04 '22 17:05 grdddj

First possibility of looking at the UI test screens in CI - https://satoshilabs.gitlab.io/-/trezor/trezor-firmware/-/jobs/2416495841/artifacts/test_ui_report/index.html

grdddj avatar May 05 '22 09:05 grdddj

Currently, my local results are 81 failed, 913 passed, 96 skipped, 7 xfailed.

The biggest part of the failures is from reset_recovery because these things are not yet implemented for R.

The second group of failures are tests asserting something in layout.lines, and they fail because we are not showing the correct things on the screen.

Some tests needed an explicit failure for model R with something like

    if client.features.model == "R":
        pytest.fail("causes freeze on the pagination screen")

because these cases would freeze the whole test pipeline. I put skip_tr only on the same spots as skip_t2 was (apart from sd_card functionality which will not ever be there), so we see what functionality is missing.

I would say the setup-UI-tests step is almost ready now and the next phase is make-UI-tests-pretty. @matejcik should I attempt to do something here, or wait for UI/UX people to give me concrete design for every screen?

grdddj avatar May 06 '22 10:05 grdddj

we probably need to add "allow fail" to the R test CI job for now? (that said the current failure looks completely wrong)

matejcik avatar May 20 '22 08:05 matejcik

we probably need to add "allow fail" to the R test CI job for now? (that said the current failure looks completely wrong)

Allow fail added in f048542. It should also fix the failing tests/device_tests/bitcoin/test_signtx_replacement.py::test_p2pkh_fee_bump, I have merged it incorrectly before

grdddj avatar May 20 '22 08:05 grdddj

fwiw I'm working on fixing the UTF issue -- it's not that Rust can't deal with UTF, but the text passed in is just right to trigger a char boundary bug in our code

matejcik avatar May 24 '22 09:05 matejcik

Force-pushed after rebase on master and to be compatible with https://github.com/trezor/trezor-firmware/pull/2277, which build on top of this

grdddj avatar Jul 08 '22 12:07 grdddj

Replaced by https://github.com/trezor/trezor-firmware/pull/2610

grdddj avatar Feb 03 '23 15:02 grdddj