indygreg
Parallel builds support
This is an attempt to support building multiple Python versions at the same time. The impetus for this is reducing the amount of work that CI does by amortizing the costs for building common dependencies.
The first 3 commits should be relatively safe to merge. 4 and 5 need a bit of testing.
Open issues:
- [x] Race condition downloading pip/setuptools when building in parallel.
- [ ] Handle 3.13 and freethreaded builds better in CI (still room for job consolidation).
- [x] macOS builds broken due to old version of GNU Make.
The 3.13 CI jobs aren't yet fully optimized and already this PR drops Linux builder total time from ~64h to ~43h, or ~67% of baseline. Not bad.
The macOS builds should be fixed. Issue turned out to be a compatibility issue changed in GNU Make 3.82 (macOS ships 3.81). Specifically commit 81f3e4babd128f6740d05b371122762924522fb6 (because I bisected it).
define syntax in GNU Make 3.82 supports an optional =. e.g. define foo =. 3.81 doesn't. I was using the = and it broke 3.81.
Ok. CI came back clean.
macOS CI total time drops from 10:27 to 8:49. 529/627=84% of baseline. Not the percentage savings as Linux. And 3.13 is still on the table for optimization. But still a nice win.
My changes in #428 will make this a little more complicated, but it's probably still worth pursuing. I'm happy to investigate it.