Generate UI screenshots in different states

[thenhnn]

States

• [x] Home screen with prime
• [x] Home screen without prime
• [x] Device settings
• [x] Network settings (including Advanced)
• [x] Toggles settings
• [x] Software settings
• [x] Driver camera
• [x] Update
• [x] All offroad alerts
• [x] All onroad alerts
• [x] Keyboard
• [x] Keyboard with numbers
• [x] Onroad
• [x] Onroad w/ map
• [x] Onroad w/ sidebar
• [x] Onroad engaged
• [x] Onroad overriding
• [x] Experimental mode confirmation dialog

[github-actions[bot]]

Thanks for contributing to openpilot! In order for us to review your PR as quickly as possible, check the following:

• Convert your PR to a draft unless it's ready to review
• Read the contributing docs
• Before marking as "ready for review", ensure:
  • the goal is clearly stated in the description
  • all the tests are passing
  • the change is something we merge
  • include a route or your device' dongle ID if relevant

[thenhnn]

I will work on reports in separate PR.

[thenhnn]

@adeebshihadeh @jnewb1 This PR is done, currently I'm thinking where to upload screenshots, to show them in report. I will create new Draft PR when PoC will be ready.

[jnewb1]

it takes 10 minutes to create this report now in CI, can we parallelize this?

Ran 20 tests in 559.315s
creating html report

OK

alternatively, perhaps we only generate a few key screenshots in CI and the rest are only run manually

[jnewb1]

html report is also empty, and a bunch of the screenshots are just the terms and conditions (ex: homescreen.png)

[thenhnn]

html report is also empty, and a bunch of the screenshots are just the terms and conditions (ex: homescreen.png)

Strange, works on my machine.

Should I make HTML report and CI pages/job summaries in this PR, or I should separate pages into another PR?

[thenhnn]

it takes 10 minutes to create this report now in CI, can we parallelize this?

Ran 20 tests in 559.315s
creating html report

OK

alternatively, perhaps we only generate a few key screenshots in CI and the rest are only run manually

I don't see any way how to parralel execution, because it will require spawn of separate Xvfb instances.

I will try implementing this as separate manual CI job.

[jnewb1]

html report is also empty, and a bunch of the screenshots are just the terms and conditions (ex: homescreen.png)

Strange, works on my machine.

Should I make HTML report and CI pages/job summaries in this PR, or I should separate pages into another PR?

the html report is automatically created and uploaded as a CI artifact

[thenhnn]

html report is also empty, and a bunch of the screenshots are just the terms and conditions (ex: homescreen.png)

Strange, works on my machine. Should I make HTML report and CI pages/job summaries in this PR, or I should separate pages into another PR?

the html report is automatically created and uploaded as a CI artifact

Right, but it's not clickable link, it's ZIP archive that needs to be downloaded and unpacked.

[jnewb1]

Right, but it's not clickable link, it's ZIP archive that needs to be downloaded and unpacked.

it doesn't need to be a clickable link in this PR, but it should still work when downloaded. currently it's an empty html

the job summary thing can be a separate pr and project

[thenhnn]

@jnewb1: and the rest are only run manually

How to do that? I can create separate "Create Full UI Report" job, how to make it disabled by default?

[jnewb1]

How to do that? I can create separate "Create Full UI Report" job, how to make it disabled by default?

just put some of the cases behind a cmd flag, such as "--all"

[thenhnn]

How to do that? I can create separate "Create Full UI Report" job, how to make it disabled by default?

just put some of the cases behind a cmd flag, such as "--all"

TEST_UI_FULL=1 will enable all test cases + offroad/onroad alerts, and it is disabled by default (check "Create Minimal UI Report")

[thenhnn]

Hmm, for some reason HasAcceptedTerms and CompletedTrainingVersion works on my PC, but not on CI...

P.S. Fixed

[thenhnn]

@jnewb1 I created new workflow full_ui_test that can be manually invoked, if I understand workflow_dispatch correctly.

[thenhnn]

Any chance to get this merged?