Update Overhaul: final QA and testing

[jotaen4tinypilot]

After the last item of the second milestone is complete, we want to do a thorough testing phase before merging the update-overhaul branch into master.

Only after all testing is done, we merge the update-overhaul branch into master. We should probably also schedule this wisely, to make sure that we have sufficient dev and support coverage in the following days of the release.

Testing scenarios

For each of the install paths, follow the setup instructions and review the testing considerations below.

Install paths

Note: Before running a test, add your Github handle to the end of the task. After you complete it, check the box for the task.

• [ ] Test new installation on Voyager device (instructions)
• [ ] Test new installation on Hobbyist device (instructions)
• [ ] Test update path from an existing (legacy) TinyPilot Community installation on a Hobbyist device (instructions)
• [ ] Test update path from an existing TinyPilot Community installation (that’s already on the new update mechanism) using any device (instructions)
• [ ] Test downgrade path from TinyPilot Pro to Community using any device (instructions)
• [ ] Test upgrade path to TinyPilot Pro from an existing TinyPilot Community installation (that’s already on the new update mechanism) using any device (instructions)

What to consider when testing?

Hardware compatibility

• [ ] Check that the installation works on both Voyager and Hobbyist device

Behaviour / integrity

• [ ] Make sure that we don’t break existing systems, e.g. by accidentally overriding existing app/system settings or configs
• [ ] Double check that the new installation places all files and folders on the system, and that they have the correct permissions
• [ ] Try out all functionality of TinyPilot to verify correct system setup
  • [ ] System > Update
  • [ ] System > Hostname
  • [ ] System > Video settings (testing one setting change is sufficient)
  • [ ] System > Logs
  • [ ] System > About
  • [ ] System > Power (testing restart is sufficient)
  • [ ] Actions > Paste
  • [ ] Actions > Screenshot
• [ ] Identify points of failure within the update flow and verify that the error behaviour is correct

Final review

Code

• [x] Do a final check that we have ported the entire behaviour from quick-install @jotaen4tinypilot
  • 👉 See https://github.com/tiny-pilot/tinypilot/issues/1078
• [ ] Do a final review of the new code, both in the bundler and in gatekeeper
  • Are there open todos? Do we need to round up anything? Are we happy with the structure and naming?

[jotaen4tinypilot]

It looks like we are about to wrap up Milestone 2 and make things happen 🎉

I’ve revisited the initial bullet points of this ticket, and tried to make them more concrete. @mtlynch @jdeanwallace Please feel free to add or adjust the items above, if you can think of any other scenario that’s not covered yet.

Thanks also @jdeanwallace for referencing this ticket from the PRs: that was pretty helpful for keeping an overview.

I’m wondering how we go about this ticket: it could make sense if two (or even three) people independently do the entire QA, just so that we have some extra safety?

As for the “tbd” regarding the Community bundles (see above): the most minimal-invasive way I can think of would be to upload a .tgz bundle built from the update-overhaul branch, and manually edit the LATEST file to point to it. That way, we don’t have to touch the update mechanism itself, and everything would exactly behave as it does for real.

[mtlynch]

As for the “tbd” regarding the Community bundles (see above): the most minimal-invasive way I can think of would be to upload a .tgz bundle built from the update-overhaul branch, and manually edit the LATEST file to point to it. That way, we don’t have to touch the update mechanism itself, and everything would exactly behave as it does for real.

Yeah, that sounds good to me.

[mtlynch]

@jdeanwallace - Is the task list ready for me to review? Just want to make sure we've merged everything over from #1046

[jdeanwallace]

@mtlynch - Yes, you can review the task list. I'm just still finalizing https://github.com/tiny-pilot/tinypilot/pull/1046 before the final round of testing.

[mtlynch]

@jdeanwallace - Okay, I've split it into individual tasks. What's gating the start of testing?

[jdeanwallace]

@mtlynch - I was hoping to merge https://github.com/tiny-pilot/tinypilot/pull/1076 & spot check the current testing instructions before getting others involved in testing.

[mtlynch]

@jdeanwallace - Okay, sounds good.

[jdeanwallace]

@mtlynch - I'm happy with the state of https://github.com/tiny-pilot/tinypilot/pull/1046 to let others start testing.

[mtlynch]

@jdeanwallace - Let's hold off on starting the tests until we fix #1081 and #1082.

[jotaen4tinypilot]

Just double-checking, since I was off: is it safe to start testing, or are there still open, ongoing issues?

[jdeanwallace]

Just double-checking, since I was off: is it safe to start testing, or are there still open, ongoing issues?

Hey @jotaen4tinypilot, looking at the project milestone view, we're still waiting on one issue that crept in before we can start the final update-overhaul test/review.

[mtlynch]

@jdeanwallace - Any blockers to begin testing at this point?

[jdeanwallace]

@jdeanwallace - Any blockers to begin testing at this point?

Nope, we can start testing! 🎉

[jotaen4tinypilot]

fyi, I have removed the point below, since I think we have this already covered separately (see first section)

Hardware compatibility

• [ ] Check that the installation works on both Voyager and Hobbyist device

Also, I’ve removed the checkboxes from the individual system features (“System > Update”, “System > Power“, etc.) and made them regular bullet points, just so that you know why the overall number of tasks is smaller compared to before.

[jotaen4tinypilot]

It all seems to be looking good so far… 🎉

• I don’t have a (working) Hobbyist device at hand, because with mine the video capture chip is defunct. I’m not sure it makes sense for me to take over the two Hobbyist test items, because I wouldn’t be able to verify whether the video setup and configs are working.
• Regarding the “test upgrade path to Pro from Community”, I realized that I actually don’t know how to do this, because I usually just flash an SD card with a disk image for setting up a Pro device. But that’s obviously not an option in this case. Is there an official way to do this, or would I just download and run Pro’s quick-install script from the private repo?
• One last-minute question occurred to me, about where to host the get-tinypilot.sh script.

[jdeanwallace]

@jotaen4tinypilot

• I don’t have a (working) Hobbyist device at hand, because with mine the video capture chip is defunct. I’m not sure it makes sense for me to take over the two Hobbyist test items, because I wouldn’t be able to verify whether the video setup and configs are working.

Okay cool, I'll take on these two Hobbyist test items.

• Regarding the “test upgrade path to Pro from Community”, I realized that I actually don’t know how to do this, because I usually just flash an SD card with a disk image for setting up a Pro device. But that’s obviously not an option in this case. Is there an official way to do this, or would I just download and run Pro’s quick-install script from the private repo?

I've sent you an email regarding the Pro installation steps.

[jotaen4tinypilot]

Pending https://github.com/tiny-pilot/tinypilot-pro/pull/541, the QA is all done.

As closing remark from my side, I’d feel confident enough to close this ticket, so I feel we have tested and considered everything quite thoroughly.