ceps icon indicating copy to clipboard operation
ceps copied to clipboard

[Exploratory] Large data (and potentially other such things) as packages

Open chenghlee opened this issue 1 year ago • 0 comments

Not sure if/how/what we'll formalize into CEP(s). But I'm starting to think about how we could use conda to deliver large data sets and other "non-code" binary blobs. Think reference genomes for bioconda packages, corpora for nltk, pre-trained models for ${your favorite new gen AI package}, etc.

We can currently do things like just generating "huge" (multi-GB) packages or (ab)using post-link/activation scripts to run wget some-suspect-url, but I'm wondering if, as a community, we can come up with more clever solutions.

chenghlee avatar Feb 07 '24 18:02 chenghlee