[question] Is it possible to apply --build=missing only to tool_requires?

[kaz8096]

What is your question?

Hi, thank you for maintaining Conan! I have a question: is it possible to apply --build=missing option only to tool_requires, and not to the regular requires?

For example, I would like to ensure that all regular dependencies are already available in the cache (or from a remote), but still allow missing tool_requires to be built locally if necessary.

This becomes especially important when migrating to environments into Ubuntu 24.04 from Ubuntu 22.04, where GCC 13 is the default. In such environments, most prebuilt binaries on ConanCenter are not usable because of mismatch gcc version, so toolchains need to be built locally — while I still want to enforce that application libraries are fully prebuilt and fetched from my own artifactory remote.

Is there a recommended way to achieve this behavior, or would this be something worth considering as a new feature?

Thanks in advance!

Have you read the CONTRIBUTING guide?

• [x] I've read the CONTRIBUTING guide

[memsharded]

Hi @kaz8096

Hi, thank you for maintaining Conan!

Thanks to you for your question! :)

I have a question: is it possible to apply --build=missing option only to tool_requires, and not to the regular requires?

No, it is not possible at the moment, there is nothing in the --build logic that differentiates among contexts. But it shouldn't be necessary.

This becomes especially important when migrating to environments into Ubuntu 24.04 from Ubuntu 22.04, where GCC 13 is the default. In such environments, most prebuilt binaries on ConanCenter are not usable because of mismatch gcc version, so toolchains need to be built locally — while I still want to enforce that application libraries are fully prebuilt and fetched from my own artifactory remote.

Those are important points.

Note that even if you update your compiler to compiler.version=13, and existing binaries exist for other previous version, lets say compiler.version=12 there are some possibilities:

• Use compiler.version=13 for your "host" regular dependencies, but use compiler.version=12 in the "build" profile for your tools-requires dependencies
• Define some fallback logic between gcc versions 13 and 12 in your compatibility.py plugin
• Define explicitly, per-package that some package you want it with mypkg/*:compiler.version=12 while you are using other 13 version for other packages.

Irrespective of the binary compatibility and resolution, there might be other approaches that you might want to take. For example, if you are already building some binaries, then, according to the recommendations in https://docs.conan.io/2/devops/using_conancenter.html, it is very often better to just take ownership of building your binaries from your own fork of conan-center-index Github repo, for the compiler versions and other configuration you might want. This is a very robust and powerful approach, that requires only a bit more of forward creation of binaries from your fork and uploading those binaries to ConanCenter.

That would solve your comment:

In such environments, most prebuilt binaries on ConanCenter are not usable because of mismatch gcc version, so toolchains need to be built locally — while I still want to enforce that application libraries are fully prebuilt and fetched from my own artifactory remote.

You basically build everything, not using binaries from ConanCenter, but your own ones. Full control, with a ton of other advantages, like being able to apply your own private patches, etc, while being totally robust against changes not under your control. And it doesn't require a large effort.

Finally, if at some point you want to differentiate between different distros with different libc versions, and different binary compatibility, the recommendation would be to add a custom sub-setting in the settings_user.yml to model de Linux distro (o even at the lower level, the libc version).

Please let me know if this helps or you have any other question.

[kaz8096]

Hi @memsharded ,

Thanks so much for the quick and detailed response — I really appreciate it! Just to clarify a bit more about my use case:

• I actually already maintain a private Artifactory with prebuilt packages for cross buildings (e.g. Android)
• However, depending on the host environment (e.g., when switching to Ubuntu 24.04), tool_requires and even Android packages themselves end up being rebuilt --- even Android and Build env don't relate with each other. That is not ideal in terms of build reproducibility and development experience.

Your suggestion to fork conan-center-index and maintain tool builds for supported environments does sound like the most reliable path forward. I’ll seriously consider going in that direction (Though I partially mirror conan-center-index already to use the recipes, but not host for build tools)

That said, I’d still really love to see more fine-grained build control in Conan, such as the ability to apply --build=missing selectively to tool_requires and/or test_requires. It would make it much easier to strike a balance between strict reproducibility and practical local development. It would be greatly appreciated if this could be considered for future improvements.

Thanks again for your help and insights!

[memsharded]

Thanks for the feedback.

However, depending on the host environment (e.g., when switching to Ubuntu 24.04), tool_requires and even Android packages themselves end up being rebuilt --- even Android and Build env don't relate with each other. That is not ideal in terms of build reproducibility and development experience.

I am not sure what is going on. Changing the "host" compiler doesn't necessarily change the package_id of the packages in the "build" context, if they are being built with the previous compiler.version.

Your suggestion to fork conan-center-index and maintain tool builds for supported environments does sound like the most reliable path forward. I’ll seriously consider going in that direction (Though I partially mirror conan-center-index already to use the recipes, but not host for build tools)

yes, the recommendation once you have to build some of your binaries is to build all of them, for all packages, including both host and build requirements, but not mixing ConanCenter binaries with your own built ones. The effort is almost the same, because you already need the infra for some, so it is just a matter of running a few more jobs, but that is cheap.

That said, I’d still really love to see more fine-grained build control in Conan, such as the ability to apply --build=missing selectively to tool_requires and/or test_requires. It would make it much easier to strike a balance between strict reproducibility and practical local development. It would be greatly appreciated if this could be considered for future improvements.

I am afraid that I still didn't fully understand the use case. Packages should be built if and when they are needed. It doesn't really matter if it is a "host" package or a "build" package, there shouldn't be any distinction. A --build=missing over tool-requires only doesn't make much sense, because that would mean that all the "host" requirements have available binaries (otherwise it will fail with missing binary error), and if the "host" requirements are available, then the tool_requires from the "build" context are actually not necessary, and in many cases they will actually be marked as "Skipped", so a --build=missing policy will not have any effect on them because they are not "missing" as they are not necessary, and they will not built. In order to really need to build some tool_requires, it is necessary to need to build some "host" requirements in the first place.

[kaz8096]

That said, I’d still really love to see more fine-grained build control in Conan, such as the ability to apply --build=missing selectively to tool_requires and/or test_requires. It would make it much easier to strike a balance between strict reproducibility and practical local development. It would be greatly appreciated if this could be considered for future improvements.

I am afraid that I still didn't fully understand the use case. Packages should be built if and when they are needed. It doesn't really matter if it is a "host" package or a "build" package, there shouldn't be any distinction. A --build=missing over tool-requires only doesn't make much sense, because that would mean that all the "host" requirements have available binaries (otherwise it will fail with missing binary error), and if the "host" requirements are available, then the tool_requires from the "build" context are actually not necessary, and in many cases they will actually be marked as "Skipped", so a --build=missing policy will not have any effect on them because they are not "missing" as they are not necessary, and they will not built. In order to really need to build some tool_requires, it is necessary to need to build some "host" requirements in the first place.

Thanks again for the thoughtful reply! — I realize I didn’t explain one important part of our workflow clearly enough. (NOTE: I’ve been speaking in the first person, but in reality, Conan is being used across multiple projects by a number of developers in my org)

In our setup (especially in CI/CD pipelines), we strictly avoid using --build=missing altogether.

The goal is to ensure that only prebuilt binaries are used, and that no unintended builds are triggered. This policy helps us maintain reproducibility and avoid accidentally building packages with incorrect or unexpected configurations.

The concern is that if --build=missing is used broadly among our teams, it might silently build not only missing tool_requires, but also upstream(dependency) packages that the project depends on, when the project has a bad configuration accidentally. This can result in binaries of the project that appear valid but are not properly aligned with the intended configurations actually. I believe this issue (contamination of bad dependency configuration projects with --build=missing use) is more likely to occur in larger ecosystems with many interconnected projects.

This is the reason why I wanted to have fine-grained controlling over --build missing for requires, tool_requires, and test_requires.

[memsharded]

In our setup (especially in CI/CD pipelines), we strictly avoid using --build=missing altogether.

The goal is to ensure that only prebuilt binaries are used, and that no unintended builds are triggered.

The concern is that if --build=missing is used broadly among our teams, it might silently build not only missing tool_requires, but also upstream(dependency) packages that the project depends on, when the project has a bad configuration accidentally. This can result in binaries of the project that appear valid but are not properly aligned with the intended configurations actually. I believe this issue (contamination of bad dependency configuration projects with --build=missing use) is more likely to occur in larger ecosystems with many interconnected projects.

Yes, that is perfect! :)

We totally recommend this, and this is why --build=missing is not the default in Conan, because the default should be to have pre-compiled binaries.

It is definitely a good strategy, thanks for sharing.

This is the reason why I wanted to have fine-grained controlling over --build missing for requires, tool_requires, and test_requires.

Still not clear the need. A --build=missing-only-tool-requires would never work, because in order to need to build tool-requires, it is necessary to need to build from sources "host" regular requires, otherwise the "tool-requires" are not needed and marked as "Skip" instead. If we wanted to consider some possible feature-request or improvement, it would be necessary to understand better the use cases and problems solved.

I also understand this is not always easy to communicate via Github tickets, or maybe you are satisfied with it being a question and "it is not possible at the moment" a valid solution. So if you are not interested in investing more time in this, we totally understand too, not a problem :) :) Just let us know, feel free to provide more feedback, close the ticket as responded, or even if you want to try to make a call, you can write us to [email protected].

Just in case, also note that in practice, it might be relatively easy to capture the conan graph info ... --format=json > graph.json and then traverse the json, filtering by context or something like that. The conan graph build-order command can also be very useful, see for example the "Products pipeline" in the CI-tutorial: https://docs.conan.io/2/ci_tutorial/tutorial.html And it might be possible to create custom Conan commands that implement custom logic too.

[memsharded]

Hi @kaz8096

Any further comment or insight about the above last comments?

[memsharded]

I am closing this ticket as resolved, but please don't hesitate to create new tickets for any further question or issue. Thanks for your feedback.