[feature] Add examples and tools to automate the creation and hosting of binaries

[andrea-masciotta-idsg]

What is your suggestion?

Hello! This page describes exactly our use case, could you please add examples and tools as suggested by this sentence?

This is the basic flow idea. We will be adding examples and tools to further automate this flow as soon as possible.

Specifically, we would like to see the script "that first conan export all the packages in your list, then conan create --build=missing them".

Thank you!

Have you read the CONTRIBUTING guide?

• [x] I've read the CONTRIBUTING guide

[memsharded]

Hi @andrea-masciotta-idsg

Thanks for your question.

I am moving this ticket to the docs repo, as this is more an issue of the documentation than an issue to solve in the Conan client.

[memsharded]

Are you aware of the local-recipes-index feature? https://docs.conan.io/2/devops/devops_local_recipes_index.html

With this feature, the flow in that page will be even simpler. So something like:

• Define a json file with only the "direct" dependencies you want to use from conan-center-index github repo, something like:

  ["zlib/1.3.1",
   "openssl/3.5",
   ....]
  

You might also be able to use version ranges here if you want

• Use a python script such as:

def run(cmd):
     subprocess.run(cmd, ...)

packages = json.loads(open("mypkgs.json", "r").read())
for package in packages:
      run(f"conan install --requires={package} --build=missing")

run("upload * .... -r=myremote)

• Define the conan remote add local <path to your conan-center-index fork>
• python myscript.py

[andrea-masciotta-idsg]

It seems a great idea!

We are trying to setup an internal automated build pipeline that builds all the binaries we need from the recipes in Conan Center using the profiles we need. If I understand correctly your suggestion, the build machine should simply:

1. Clone our fork of the Conan Index
2. Call the script you sketched, passing '-pr=<our_profile>' to 'conan install' for each of our profiles
3. Call 'conan upload * -r=myremote'

Can you confirm?

Some more questions:

• Do you suggest that we automate the merging of the updated from the public Conan index into our fork and then retrigger the pipeline to build any new recipe or keep the process manual?
• Which configuration should we set in our Conan clients to consume the packages built in this way? Should the clients have some knowledge about our fork of Conan index or would it be enough to make them point to myremote?
• What are the pros and cons of disabling conancenter as remote on our Conan clients, keeping our internal one only?

[memsharded]

Can you confirm?

Yes, exactly :)

Do you suggest that we automate the merging of the updated from the public Conan index into our fork and then retrigger the pipeline to build any new recipe or keep the process manual?

It depends. This can depend on many factors, like how much do you prioritize stability over fast moving to the very latest versions. Also on the available CI resources, on the available developers time, etc, etc.

I'd probably go (without knowing your constraints or requirements, just some general subjective opinion) for something hybrid, that is, have the process almost automated, but trigger it, control and supervise it regularly, to avoid problems at the least expected times, for example when about to do a releases, a nightly update of the ConanCenter dependencies could break because some new changes in a recipe, in a way that is difficult to detect and debug. So maybe the process is to be triggered manually at the beginning of the sprints, when there will be enough time to evaluate that everything is working as expected.

Which configuration should we set in our Conan clients to consume the packages built in this way? Should the clients have some knowledge about our fork of Conan index or would it be enough to make them point to myremote?

If you are building for a known set of platforms, and you have some common defined profile files, (that you can manage with conan config install/install-pkg, then typically users consuming the packages only have to point to your own remote containing the package binaries from your build of your fork to use them. As a general rule, the local-recipes-index and the package creation process from a conan-center-index fork is a "package generation" strategy, so developers shouldn't typically be aware of this, for them the packages come from a remote in the same way they could come from ConanCenter directly, but they don't need to be aware of the details of how those packages were produced.

What are the pros and cons of disabling conancenter as remote on our Conan clients, keeping our internal one only?

Pros:

• Full control, there is absolutely nothing that can break your build without your explicit trigger. If you keep depending on ConanCenter, new versions published there, new recipe changes, etc. could affect your builds. It is true that you can use lockfiles to keep locked to an exact set of dependencies, but without these lockfiles it is easier to break
• Less binary compatibility issues. ConanCenter uses a relatively old glibc to achieve a wide binary compatibility. But if you build your own binaries, you have full control over that, can even define custom settings_user.yml to model different binaries for different Linux distros. Also better compatibility with your own proprietary packages
• Increased security. We put a lot of effort in the security of ConanCenter, but nothing compares to fully owning the full Software Lifecycle. Depending on your developer resources, you could manually audit every new version upgrade befores it enters your production.
• Less likely of disruptions due to network issues, or possible ConanCenter maintenance windows.
• Better alignment of package customizations and versions to be used. Like changing the default_options for packages to suite your needs simplify a lot of operations (less information necessary in profiles, for example, simpler command lines, etc)
• Full ownership of the pipeline means instant reaction time, you can patch, fix, change or customize any package there instantly, without needing to wait for a review process on the public ConanCenter, than can take much longer time.

Cons:

• Might need a bit more time and resources of CI. Typically, as the CI is already needed for your own builds, it means no extra infra or maintenance effort, and the resources needes are relatively small compared with the very frequent builds that happens for your proprietary code builds
• Might sometimes live in the not very latest changes or fixes of ConanCenter, and some recent bugfixes of packages might not be incorporated in your products. But if your products have enough testing and QA, most likely this is not an issue in the vast majority of cases.