[feature] Make git the source of truth for recipes

[tsondergaard]

What is your suggestion?

Hi! I've spent some time evaluating both Conan and Vcpkg and there are many things that I like about both projects. So far I like Conan a little better overall, so I want to provide feedback to the Conan projeect on things that I think it should borrow from Vcpkg. One of the things that I think Conan should take from Vcpkg is to keep the source packages (recipes) in git and stop publishing them to a separate repository. It sounds like a small thing, but I don't think it is. All development teams have git repositories and are comfortable with the precise tracking of every little change. It is our source of truth. But for Conan recipes there is an intermediate step where recipes have to be published to a Conan repository and we now have to trust a process rather than a tool that the published packages are correct. We also do not have a commit history for the repos published to the Conan repository, of if we do it is certainly not as accessible to developers as git log.

Have you read the CONTRIBUTING guide?

• [X] I've read the CONTRIBUTING guide

[memsharded]

Hi @tsondergaard

Thanks for your feedback.

I think you would be interested in https://github.com/conan-io/conan/pull/13930, it sounds that it is what you are describing. We are aiming to release it in next release, but can't guarantee, as it is complex and has some risks, but we are on it.

Said that, I'd still like to advocate for the server model and encourage not to discard it. As part of JFrog we have been able to learn a ton about package management in general and modern devops, these would be some important points:

• Basically all language package managers and system package managers work with a server-client model and not a git source model. Language (Maven for Java, NPM for JS, PyPI for Python, Crates for Rust) ones, system (apt, yum, etc), others similar tools (docker), all work with a server-client model. In some cases, having the binaries is critically important, it doesn't make sense to have to build from sources all the time. We believe that this is very important too for C and C++. Languages like golang that had a pure git model was a big pain and it had to evolve to a modules model, using also the server-client model, even if it is not distributing binaries. Same with crates for rust, it doesn't distribute binaries, yet it uses the server-client model as main paradigm.
• There are many reasons for this server-client model, but the limitations of source approach is important. vcpkg focus mainly on a flat (no-versions, or just 1 version in the repo), and even when versions were added, it doesn't have the same power to implement package-management important features like version-ranges, version resolution, conflict detection at scale. But probably one of the most important is decentralization of sources. The source approach can only work for open source third parties centralized in one repo, which is what vcpkg does, and what https://github.com/conan-io/conan/pull/13930 is leveraging from the conan-center-index github repo containing all ConanCenter recipes. But for the case that users want to start creating their own packages for their own code, it can become quite messy. As an example I recently bumped into the azure-sdk libraries (part of Msft too), and look at their CMakeLists.txt: https://github.com/Azure/azure-sdk-for-cpp/blob/main/CMakeLists.txt. This is super-intrusive, while in Conan we work hard so users don't have to modify a single CMakeLists.txt line.
• So the decision if it is just for open-source third party or for your own packages too is very important. The moment users have to create their own packages for their own source code, there will be possible different people, from different source repos, creating different versions (because your own source code evolves way more, third parties can be frozen, but your own code not), then the server-client model becomes critical, the source approach is simply not maintainable.

But for Conan recipes there is an intermediate step where recipes have to be published to a Conan repository and we now have to trust a process rather than a tool that the published packages are correct.

You already have to trust the process. The moment you are building things from source to create your final product that you release or sell you have a process. You will probably have tests, CI that build things, storing the binaries somewhere for the releases... The devops practices with server-client model allow to structure and standardize this process a bit more. The build of a given library with the upload of the different binaries for the different platforms and versions to a server becomes a very valuable process. Knowing that every developer and CI can just install mylib/1.2.3 pre-compiled binary with the same process and commands in every platform, for every version, and being mylib from any team in the organization, in their machines and link against it, test it, etc. is very valuable.

The process allows a "fail fast" and mitigate many of the problems that comes from source. When a library is being changed, the process can quickly build the package from that change, or fail if not possible. This allows consumers of that package to test themselves without having to rebuild over and over the same lib, or having to implement their own self-made binary caching and management. The source repo model needs to work a very massive CI that builds the whole repo periodically.

We also do not have a commit history for the repos published to the Conan repository, of if we do it is certainly not as accessible to developers as git log.

You can have almost the same with revision_mode = "scm" in your recipes, and then the revision will be the git commit, a conan list mypkg/1.0#* will list your revisions in chronological order (time to upload to the server). We are also going to leverage this to implement a new command that clones a repo at a specific commit based on the package reference.

TL;DR: For a third-party open source model, the source repo can be ok, and this is the reason we are enabling that for repos with recipes like conan-center-index in https://github.com/conan-io/conan/pull/13930, given that building from source is not a big issue (not using tons of third-parties, or using those that can be built fast enough). For other cases, I'd strongly recommend to use a server-client model. At least for Conan, and even with https://github.com/conan-io/conan/pull/13930, it would be counter productive, probably working uphill.

[tsondergaard]

Thanks @memsharded for the comprehensive answer, and #13930 does look interesting. Looking forward to it, regardless of the release that it will be included in!

[memsharded]

The "local-recipes-index" has been already released some weeks ago in Conan 2.2, and announced in https://blog.conan.io/2024/04/23/Introducing-local-recipes-index-remote.html.

I think we can close this ticket as solved, if there are any further question, or any feedback while using the "local-recipes-index" new feature, please create new tickets. Thanks!