[develop2] Consider re-enabling of file download cache

[memsharded]

We are removing the "file download cache" in 2.0, because of some complexities, missing development in the 2.0 cache concurrency and lack of consistent feedback of the performance gains of the "file download cache".

In order to consider re-enabling the file download cache we need reports for different users about:

• What performance gains are being observed by enabling it
• What is the most important perceived value
• Is it being enabled for developers machines, CI, or both? For all OS? is the performance similar in all OS?
• Is it necessary to decouple caching of internet files (sources downloads) and caching of files coming from a conan server (like a conan_package.tgz from a Conan package binaries)

As we are no longer implementing this kind of new features in Conan 1.X, and the feature has been removed in 2.0, I am closing this issue, and opening a new one for 2.X scope from this comment.

Originally posted by @memsharded in https://github.com/conan-io/conan/issues/9428#issuecomment-1186622504

[madebr]

I heavily use this caching whilst creating and/ormodifying recipes for 3rd party dependencies, e.g. for those on conan-center-index. It is a huge time saver + is also useful to keep working on a recipe when the internet connection is flaky.

[memsharded]

I heavily use this caching whilst creating and/ormodifying recipes for 3rd party dependencies, e.g. for those on conan-center-index.

So that means, that you are only interested in the caching of downloaded sources from the internet, and not the rest of the features.

This use case might be also partially addressed by the recent file:// support in get/download. Not exactly the same, it still requires some manual work, but would be something to consider until this is decided.

[MTomBosch]

Hello, for us the performance gain is not the only argument why it is required. For us as important as the performance is the reduction of traffic from our JFrog SaaS instance towards the CI/CD build machines. Without the downloac cache we will need to pay a 5 digit number $ per year more than right now with the download cache. If there is no download cache or something similar to reduce the traffic from Artifactory towards the CI/CD build machine we will not be able to move to Conan 2.0 at all. So please get it working for 2.0 again or provide something similar.

[jellespijker]

We also rely heavily on caching downloads between pipeline runs on our GitHub Actions. https://github.com/Ultimaker/Cura/blob/d69819a24e89af2b35266b25c5bc91561048bb78/.github/workflows/conan-package-create.yml#L80

Mind you I'm not sure if this is really needed in our case, but since I can't see the actual bandwidth usage in our Artifactory Enterprise Edition it seemed like the most logical thing to enable as some defensive bandwidth limiting strategy.

[memsharded]

Thanks for the feedback, we will look into it before 2.X.

[crsib]

Audacity uses storage.download_cache to make offline builds possible on Linux. Removing it will be definitely a deal breaker for us.

[memsharded]

Thanks @crsib

This is exactly the feedback we wanted. We were concerned that the cache was not proving to be as performant as expected, which was the initial goal. It seems that the value provided was different from performance, but bandwidth, costs, network, etc.

I will give it a try later to re-enable it for the next beta.4

[memsharded]

Cache re-enabled in https://github.com/conan-io/conan/pull/12138, will be in beta.4