conda-forge
The accelerating bitrot of `osx-64` and falling download numbers
Dropping support for anything related to the size of the pool of potential users is always a contentious topic in conda-forge. We've taken forever to move to cos7, move to macOS 10.13 (much less 11.0), and we haven't come to a conclusion on ppc support either.
To a degree, this is healthy - we shouldn't drop support at the first opportunity - but on the other hand, we need to be pragmatic enough to pull the plug on stuff if there are either no more users, or we cannot provide a baseline level of packaging quality (bugfree-ness, security posture).
I don't/didn't think that osx-64 would be up for debate in the near future, but on the other hand, a lot of signs are accumulating. Anaconda dropped Intel Macs this summer, many other big packages (e.g. upstream pytorch) have long done so as well (and we're struggling quite a bit with v2.9 due to this). There are many more stories of upstream osx-64 support bitrotting quickly, but those were all around non-central packages that we can mostly work around (like we do for the whole google-sphere of packages like abseil, grpc, protobuf, etc.).
However, what caught my eye was that cryptography will drop support for osx-64 in their next release: https://github.com/pyca/cryptography/issues/13520. Since osx-64 is still a tier-2 platform in Rust (the guts of cryptography, ever-increasingly), I expect us to be able to keep things going for a while longer, but this really is a clarion call IMO. If - for whatever reason - we lose the ability to publish security-critical library updates on a given platform, that's the absolute latest point where we have to declare it dead and gone.
Other relevant data points:
- The last pre-M1 hardware was sold into 2020 AFAICT
osx-64is still supported until MacOS 26 (Tahoe), which is the last one to support (some) Intel macs- MacOS 26 will be EOL upstream in Sept. 2028, if the recent patterns on these dates hold
- Microsoft is deprecating the osx-64 CI agents (while still far behind in delivering osx-arm64 agents); once that happens, we'd have to switch to cross-compiling
osx-64fromosx-arm64. In theory this should be exactly what we've been doing for years the other way around (i.e. cross-compilingosx-arm64fromosx-64), but at our scale, I wouldn't be surprised if issues arise from having to do that switch.
Download numbers (as taken through the proxy of cryptograpy, which, as any proxy, is far from perfect) have also been falling off a cliff in the last year (these are only osx-* downloads from PyPI, courtesy of the cryptography maintainers)
Conda(-forge)'s own data is tainted by the downloads from our own CI, see: https://github.com/anaconda/anaconda-package-data/issues/64
Overall, I think we could probably squeeze another 2, 3, 4 years out of osx-64 to the degree that stuff builds without issue (assuming the cross-compilation switch works ~smoothly, and key packages stay compilable with minimal patching), but I wanted to document the situation as I was looking into it a bit this week. And perhaps people have other opinions or additional data points about the whole thing.
Thank you for opening this, @h-vetinari. I agree that documenting the early signs will be critical to our decision making in the future. The download data for cryptography is informative enough, imo.
My personal threshold for "let's talk about this for real" is when Azure (finally) offers osx-arm64 runners (hopefully before the year ends) and we can offer that platform by default in staged-recipes and feedstocks. Until then we are in a difficult situation to make any calls.
Also agree that https://github.com/anaconda/anaconda-package-data/issues/64 is important for us to make this type of decisions.
This is always a hard call. Overall, I'm sympathetic to keeping things working for as long as they can reasonably be kept working. We need to also consider that with decisions such as cryptography's, conda-forge may be one of the last options for some users of these platforms.
Of course, support for runners is critical here. However, my understanding is that even if Azure discontinues osx-64 runners, we'd still be able to cross-compile from osx-arm64 — and that wouldn't be that different from how we're supporting osx-arm64 now. In fact, I'd expect the osx-arm64 runners to be more powerful, and therefore better suited for cross than the other way around.
So while I agree that we should be ready to eventually drop support for osx-64, I think it's reasonable to keep it for as long as we: a. have some kind of working build infrastructure, and b. keeping things working doesn't require significant effort from the maintainers. I'm not a macOS expert, but I can imagine that the support for osx-64 will largely remain working for as long as linux-64 and osx-* are supported by projects.
It seems to me like there's a state in between "let's drop it completely" and the status quo, and it's time for osx-64 to move to a state similar to ppc64le where it's up to maintainers to drop it from individual package builds.
many other big packages (e.g. upstream pytorch) have long done so as well (and we're struggling quite a bit with v2.9 due to this)
This is a bad state for multiple reasons. Wasted precious development effort is one. Another is that PyTorch 2.9.0 has been out for over a month now, and it still not being available on conda-forge is bad for end users and for conda-forge adoption/retention (an up-to-date PyTorch is critical for a significant fraction of deep learning developers). It seems like a no-brainer to me to move ahead with publishing 2.9.0 pytorch packages for the other platforms and forget about osx-64. Users who still need osx-64 are not performance-sensitive and will be fine with 2.8.0.
Since we don't have osx-arm64, it's critical that we keep building for osx-64. Even if we don't build for osx-64 for pytorch, the downstreams will need a osx-64 pytorch to build the downstream for osx-arm64. Also we only test osx-64 and dropping it will make our testing bad in osx- platforms. The bigger picture is that dropping osx-64 without native osx-arm64 runners will make our osx-arm64 support much much worse.
For the record, looks like we can't build PyTorch on osx-arm64 either anymore, so the PyTorch build problem became irrelevant here. My "best of 4" timed out around 80% of compiling libtorch.
The bigger picture is that dropping osx-64 without native osx-arm64 runners will make our
osx-arm64support much much worse.
It may be more useful to put this in terms of constraints/requirements (e.g., "macOS test coverage for arm64 must remain the same or improve") rather than an assessment like this. If for example, for the PyTorch feedstock, the solution is to hook up more powerful hardware to overcome the current hurdle, than hooking up arm64 rather than x86-64 will likely be (a) better rather than worse for arm64 test coverage & resulting package quality, (b) effort that's more useful going forward, and (c) reduce build time by a lot because arm64 cores are so much faster.
Sure. Here's a list
- test coverage must remain the same or improve for at least one macos platform.
- ability to building downstreams for osx-arm64 must not decrease.
Your example solution for pytorch feedstock of hooking up more powerful hardware will satisfy 1, but not 2. (unless we hook up every pytorch downstream to use the hardware)