community
community copied to clipboard
dask
Cleaning up old nightly conda packages
Having support for dask nightlies has been wonderful from a development standpoint
Recently we noticed that we are getting quite a lot of them tracking back over the years (back to 2021!)
As a result this means there are a lot of packages in the dask channel
This can make the channel repodata get quite large (and unfortunately a bit unwieldy)
So am wondering if we can clean this up a bit. Namely would it be reasonable to remove all nightlies after making a release?
Presumably users only care about nightlies newer than the last release. So dropping the old nightlies shouldn't matter much. Perhaps we can do this in an automated GHA script when a git tag is detected
Would be interested to hear thoughts on this 🙂
Thanks for the issue @jakirkham. Cleaning up old nightly builds seems very reasonable to me.
Presumably users only care about nightlies newer than the last release
I don't use these builds myself, but that seems sensible. To my knowledge the main consumers are various RAPIDS-related users / projects (I could be wrong about that), so if you / those folks are happy with that policy, then 👍
Thanks James! 🙏
Cool we can look into this then
Ok it might be worth double checking at Coiled. Recall hearing these were used previously ( for example: https://github.com/coiled/benchmarks/issues/311#issuecomment-1239477452 ). Though don't have much visibility over there
Fair point. Though I think things have changed in that repo since then (cc @hendrikmakait in case you happen to know if the dask conda channel is used)
you might want to check with the scientific-python people, as far as I can tell they have something similar working for scientific-python-nightly-wheels and the upload-nightly-wheels action.
cc @matthewfeickert, in case you have any hints
Dask nightlies sound pretty great to me. :) If the Dask team is interested, either give me the GitHub handle of a team member to work with and I'll setup a Issue on https://github.com/scientific-python/upload-nightly-action, or have a team member take a look at https://github.com/scientific-python/upload-nightly-action/issues/87 and then self start a similar issue.
edit: Ah, I only read the last comment before posting. I guess the Dask team already has nightlies and wants a way to clean them up? If so, we have a system that works for https://anaconda.org/scientific-python-nightly-wheels using https://github.com/scientific-python/upload-nightly-action/blob/cb172cd73e045bb8b9fdaf271a183ffe4b5814d3/.github/workflows/remove-wheels.yml, but that's currently (c.f. https://github.com/scientific-python/upload-nightly-action/issues/62) only for scientific-python-nightly-wheels.
Opened https://github.com/dask/dask/pull/11294 with a first pass at what these changes could look like
Also given recent inactivity on dask-sql (https://github.com/dask-contrib/dask-sql/issues/1344), think we should also be good to outright purge dask/dask-sql now
