Python-Apple-support
Python-Apple-support copied to clipboard
Optimise and improve GitHub Actions workflow
It currently takes almost 90 minutes to do a CI build of the support packages. This is at least partially due to duplicated build effort. We use a simple matrix build to target each build platform (macOS, iOS, tvOS, watchOS); however, building iOS requires a build of macOS, so macOS is effectively built 4 times; this also means there are duplicated maOS builds of bz2, xz and OpenSSL.
Describe the solution you'd like
An optimised GitHub Actions workflow for builds where the macOS build step is a pre-requisite of the iOS/tvOS/watchOS targets.
If we're making changes to GitHub Actions, it would also be desirable if:
- The various build, install, and merge logs were attached as artefacts to the build, so that they can be easily accessed afterwards
- The build process included a "sanity check" that confirms the count of binary modules is what is expected. This will help avoid problems like #167.
It currently takes almost 90 minutes to do a CI build of the support packages
It's even worse than that, because we can only run 5 concurrent macOS jobs per GitHub account, and we're currently spawning 16 (4 Python versions * 4 platforms). So even if each job was reduced to 30 minutes, the total build time would still be over 90 minutes (30 minutes * 16/5), during which time all macOS jobs in Briefcase and Toga would be blocked, as happened yesterday.
Other ideas:
- Since we're not currently using the watchOS and tvOS platforms, they might as well be disabled.
- If it's anything like the Android build, OpenSSL is probably the most time-consuming component. Could we build that only once and then share it between all the Python versions?
Even though we're not using watchOS and tvOS at present, I'm hesitant to disable them; if they're not being built regularly, then the patches that have been written to support those platforms are going to atrophy rapidly.
The dependency builds are definitely fertile ground for optimization, though. BZip2 and XZ only rarely change; they definitely don't need to be built on every CI build. OpenSSL is updated more often - but the OpenSSL build can/should be decoupled from the Python builds (as the release cadence is completely decoupled).
It might be worth exploring how we can use Releases for multiple products - if tagging bzip2-1.0.5
produced a "BZip2 1.0.5" artefact, and a Python build pulls down that artefact as a pre-requisite, we could optimise a lot of the build process. It would also provide an entry point for producing the binary wheel that is needed to support some third-party packages (as libFFI and OpenSSL are pre-requisites of some packages).
This has now been resolved; the macOS support package is built from the official release artefacts, and the iOS/tvOS/watchOS builds have their dependencies offloaded to the https://github.com/beeware/cpython-apple-source-deps repository.