source code caching?

[beckermr]

I am hitting cases where source code from non-github/pypi sources is being either updated but with the same filename or is gone completely. Have we considered caching source code tarballs or enabling CF to do this if desired?

[jakirkham]

We have in some cases reuploaded these artifacts on the GitHub release of the feedstock.

[beckermr]

Interesting. I was thinking of something more automated where if the conda-forge.yaml has the right setting and the PR is a version bump, it downloads the artifact, uploads to someplace (could be the github release), and then makes a change to the recipe and makes a new PR.

[jakirkham]

There was a separate discussion about using split packages to handle this. So one artifact users would produce would be the source.

[isuruf]

@scopatz, we have access to a google drive through NumFOCUS right? Is there a limit on the space?

[scopatz]

we do, no limit as far as I know

[isuruf]

@scopatz, can I get access?

[bgruening]

Bioconda is using https://depot.galaxyproject.org/software/ (https://github.com/galaxyproject/cargo-port). Every tarball is mirrored. Are you considering to do this for every package as we do in Bioconda?

[isuruf]

Are you considering to do this for every package as we do in Bioconda?

yes.

How is this server hosted?

[bgruening]

University hosted infrastructure in PSU.

[scopatz]

I am away for a week, but feel free to ask around

[sodre]

@isuruf, I think it would be awesome if we could store the tar-balls directly in anaconda.org under a different channel, e.g. conda-forge-source so we don't explode the repodata.json.

At the end of the day, we are looking for some Object Store, and I think we have a great one with anaconda.org that has an API we are all familiar.

[isuruf]

@sodre, you can only upload conda and pypi packages to anaconda.org.

[sodre]

We could wrap the tar.gz or source files in conda rather easily, wouldn't that satisfy the requirement?

I raised a similar issue to conda-build team about packaging models for Data Scientists. This is not that different.

c.f.: https://github.com/conda/conda-build/pull/3529

[isuruf]

True. Do you have some script to do that? We can hook it up to conda-forge-ci-setup.

[isuruf]

That wouldn't work, IMO. It'll be harder to replace the file in the recipe

[sodre]

Let me run a couple of tests...

[jakirkham]

Where did we land on where the sources should be stored?

[jaimergp]

@wolfv has mentioned that it would be possible to store the source(s) as part of the OCI mirroring strategy. I don't know how tricky that is, or which part of the pipeline is responsible for that, but leaving this info here in case it's needed!