[feature] source downloads and the cache

[paulharris]

A few things

• I noticed that conan.tools.files.get() doesn't seem to use the download cache. So I reverted back to the conans get tool. I was trying to upgrade away from the old conans tools.

• I'd like to be able to tell conan to "download" from a folder on my disk, ie url: file:/// if necessary

• Do not use the internet at all if the source file is in the download-cache folder. I had VTK-9.1.0.tar.gz already downloaded. When I do "conan create" it download and unpacks quickly. When my internet was disconnected, conan was not able to execute the create command - hitting the internet for some reason?

[memsharded]

Hi @paulharris

Do you provide a checksum for the get() call? Downloaded files will only be cached if they provide a checksum in the recipe, which is in any case a strongly recommended practice.

I'd like to be able to tell conan to "download" from a folder on my disk, ie url: file:/// if necessary

Yes, this is already a filed request (https://github.com/conan-io/conan/issues/8846), @jcar87 was probably going to have a look

[paulharris]

Re checksum: yes, a sha256 one. The file was in the cache already. I worked around the problem of broken internet by manually untaring the source where it needed to go. The filename isn't the checksum of the content, but you can guess from the datestamp on the file, and then checking with "tar tfv"

[paulharris]

I'm watching it download msys2/cci.latest, the conan_package.tgz

it seems to use the download cache sometimes, maybe just not for all the downloads?

[memsharded]

I think it should be caching the conan internal .tgz files, and the users ones, only if they provide a checksum.

It might be possible that it detects when there is a mismatch (a new recipe revision, for example, because something changed in ConanCenter package), and then re-download the new revision artifacts

[paulharris]

I see msys2 has a checksum for the download (and it hasn't changed) That recipe HAS changed recently, 9 days ago, but conandata.yml hasn't changed since the start of the year.

Surely the artifact would be the same hash? Or could there be multiple copies of the same msys2 tarball in the downloadcache? ie the hash isn't the file content hash, what is it?

[crsib]

I understand that this is a low priority issue, however there are steps to reproduce it using CCI recipe:

mkdir conan-dl
cd conan-dl
export CONAN_USER_HOME=$(pwd)
conan config init
conan config set storage.download_cache="$(pwd)/download_cache"
conan install gnu-config/cci.20210814@ --build
ls -l download_cache

Recipe downloads a file of 77,533 bytes using conan.tools.files.get(). There is no such file in the download_cache however.

[crsib]

Looking through the code, it appears that the "new" download() uses tools.files.download:download_cache instead of conan config set storage.download_cache="$(pwd)/download_cache". It seems that tools.files.download:download_cache is undocumented, but it is used in unit test

[paulharris]

The issue seems to be that the source seems to be re-downloaded every time the recipe changes. Perhaps the hash is a function of the recipe and filenames, rather than the hash of the download itself? I can understand the meta-hash would be required for things like patches from recipes, but please can we use the sha1 hash for the source downloads, when they are provided by conandata.yml ?

[AbrilRBS]

Moving this to the docs repo to add some documentation for core.download:download_cache