Add runtime tests for final Briefcase outputs
Briefcase has an extensive unit test suite; however, there's no formal test that the final product actually "works". Errors in templates, or changes in platform tooling occasionally cause errors
Describe the solution you'd like
Additional steps in the build matrix that verify that Briefcase will actually generate working output for each of the supported platforms.
Ideally, we would also have automated testing that the apps start and shut down; but as an initial step, just verifying that we can generate an app would be an improvement.
Describe alternatives you've considered
Nothing; but anything will be better than 100% manual testing (the status quo).
Additional context
#484 was an initial attempt at this.
The same tests should also be performed on the initial app template (beeware/briefcase-template), plus the platform templates (beeware/briefcase-macOS-app-template, etc)
A couple of recent breakages which would have been caught by a simple build test:
- briefcase-template#32: AppImage configuration was mislabeled in pyproject.toml.
- #816: Flatpak CPython installation was broken by changes in the support package mechanism.
Actually, I see there are build tests in the CI now, but these problems weren't caught for other reasons:
- briefcase-template #31 and #32: Template updates should trigger a Briefcase build test. But even if they don't, this should still have shown up in the checks of the next Briefcase PR (probably #803). I guess the reason it didn't is that the CI uses
--no-dockerfor AppImage builds. - #816 was missed simply because there's no CI build test for Flatpak.
+1 to both of these.
In addition to the Flatpak test, we should add a test for iOS and Android - neither of those are being tested either.
We should also modify the tests to actually run the binary that is produced - we've had some regressions where Briefcase will successfully generate an app, but the app won't start. The complication here is that we need a "unit test mode" for the generated stub apps - at present, iOS and Android apps in particular don't have a concept of apps "stopping" (other than a crash), and the log streamer doesn't catch those exit conditions. Xcode has the concept of a unit test target - an app that doesn't follow the usual app lifecycle; we might be able to exploit this. I presume Gradle will have something similar.
For taking this one step further and running tests within the output executable, see #399.