conan-io
[question] build-order-merge multiple files containing the asame package id with different contexts
What is your question?
Hi
I'm trying to use packages from the conan-center-index following the guideline here: https://docs.conan.io/2/devops/conancenter/hosting_binaries.html
To keep the production environment as stable as possible, I'm also following the guideline here, using multiple repositories (packages, products, develop): https://docs.conan.io/2/ci_tutorial/products_pipeline/full_pipeline.html
Furthermore:
- I have debug builds, where the
build_typeis Debug in the host context, and Release in the build context - I require packages, that must be present in both the host and the build context. E.g. Qt, where uic, moc, rcc tools are used in build-time.
Now, during the execution of the packages pipeline I have build order files for debug and release builds. The important differences are:
-
Release
- the build order file created with
-pr:h default -pr:b default - qt/5.12.6#abc123:def456 required to build
- qt/5.12.6#abc123:def456 is in host context
-
build_argscontains--requires=qt/5.12.6
- the build order file created with
-
Debug
- the build order file created with
-pr:h default_debug -pr:b default - qt/5.12.6#abc123:def456 required to build (same as in Release build order file)
- qt/5.12.6#abc123:def456 is in build context
-
build_argscontains--tool-requires=qt/5.12.6
- the build order file created with
After the two build order files are merged (--reduce is applied) the resulting build order file contains a requirement for Qt like that:
- qt/5.12.6#abc123:def456 required to build
- qt/5.12.6#abc123:def456 is in host context
-
build_argscontains--requires=qt/5.12.6 - with two filenames:
build_order_release,build_order_debug - In the
profilessection:-
build_order_releasecontains-pr:h default -pr:b default -
build_order_debugcontains-pr:h default_debug -pr:b default
-
My question is: how to figure out the required commandline arguments from profile section in the merged build order file?
- If I concatenate all the profiles:
-pr:h default -pr:b default -pr:h default_debug -pr:b defaultthen the resulting build_type in the host profile depends on the order of the concatenation. - Doing some kind of matching between the name of the profile (
build_order_release) and theinfo.settings.build_type = Releaseis kind of funky, what I like to avoid.
Any help, suggestion is highly appreciated
Have you read the CONTRIBUTING guide?
- [x] I've read the CONTRIBUTING guide
Hi @bukulin
Thanks for your question. It is indeed a good observation.
The conan graph build-order provides the complete information of all the different profiles combinations that leads to a certain package_id.
But that doesn't mean they have to be combined. That means that any of those combinations will result in the same package_id, and it is necessary to pick only one of them. Just 1 build with 1 set of arguments is necessary to build that package_id, and the others are discarded.
This is done to let users choose which one, because from their side, not all combinations are the same. For example, one package could be built both in a Windows machine natively or cross-built from a Linux one, resulting in the same binary with the same package_id. But the user might prefer to prioritize the cross-build because those Linux machines are faster, for example
Please, let me know if this clarifies the issue.
Hi @memsharded
Thank you for your quick reply! I'll go on with one of the profiles.
However, I think there are unfortunately still some kind of uncertainty. The simplified version of my build order file looks like that:
{
"order_by": "recipe",
"reduced": true,
"order": [
[
{
"ref": "qt/5.12.6#1ddd5ee70018c0b5a0f7ad6f324e4adf",
"depends": [],
"packages": [
[
{
"package_id": "c2c01d27ce730ee26cddbc8d8475f362ea0770ac",
"prev": null,
"context": "host",
"binary": "Build",
"options": [],
"filenames": [
"build_order-linux-default",
"build_order-linux-default_debug"
],
"depends": [],
"overrides": {},
"build_args": "--requires=qt/5.12.6 --build=qt/5.12.6",
"info": {
"settings": {
"os": "Linux",
"arch": "x86_64",
"compiler": "gcc",
"compiler.cppstd": "17",
"compiler.libcxx": "libstdc++11",
"compiler.version": "13",
"build_type": "Release"
},
"options": {
...
},
"requires": [
...
]
}
}
]
]
}
]
],
"profiles": {
"build_order-linux-default": {
"args": "-pr:h=\"default\" -pr:b=\"default\""
},
"build_order-linux-default_debug": {
"args": "-pr:h=\"default_debug\" -pr:b=\"default\""
}
}
}
If I choose build_order-linux-default profile, where -pr:h=default (where build_type is Release), then the resulting build will be a Release build, as it is requested in order section.
But if build_order-linux-default_debug is choosen, where -pr:h=default_debug (where build_type is Debug), then the resulting build type will be Debug, because build_args contains "--requires=qt/5.12.6" opposed to --tool-requires=qt/5.12.6 that were in the original (unmerged) build order file.
So, what you suggested here:
That means that any of those combinations will result in the same package_id, and it is necessary to pick only one of them. Just 1 build with 1 set of arguments is necessary to build that package_id, and the others are discarded.
I think is true only for the unmerged build order files. Some pieces of important information might be erased during the build order file merge. The original unmerged build order files are consistent by their own:
- The build order file for the release version requires release host profile and qt to be built for the host context
- The build order file for the debug version requires release build profile and qt to be build for the build context
Please let me know if I misunderstood something. (What is highly possible)
Thanks for the clarifications, that helped. I think you might be right, there might be an issue in the merging of build-orders.
I'll try to reproduce with a test to investigate it.
You were right, I had forgotten how it was implemented.
At the moment the merged build order for 1 package_id, even if there are multiple filenames to represent the different build-order files that contributed to this package_id, only the first one is the one represented in the current information.
It will look something like:
{
"package_id": "efa83b160a55b033c4ea706ddb980cd708e3ba1b",
"context": "build",
"binary": "Build",
"filenames": [
"bo_debug",
"bo_release"
],
"build_args": "--tool-requires=dep/0.1 --build=dep/0.1",
"info": {
"settings": {
"build_type": "Release"
}
}
}
"profiles": {
"bo_debug": {
"args": "-s:h=\"build_type=Debug\" -s:b=\"build_type=Release\""
},
"bo_release": {
"args": "-s:h=\"build_type=Release\" -s:b=\"build_type=Release\""
}
}
So the package_id context informationin "context": "build", is the one that matches the filename=bo_debug, and the one that matches the build_args = --tool-requires=dep/0.1 --build=dep/0.1.
So that first filename is the only one that should be used, in this case the bo_debug one that corresponds to "args": "-s:h=\"build_type=Debug\" -s:b=\"build_type=Release\"".
At the very least, I'll add some clarifications and explanations to the docs about this behavior.
If we wanted to support the full freedom to choose which profile to use, then it would be necessary to implement some other extra information:
- A new
build_args_per_configuration(a better name for sure), that is a dictionary between filenames and thebuild_argscorresponding to thefilename - A new
context_per_configurationthat does the same, a dict for eachfilenameto specify the correct context.
Maybe it would be better to convert filenames to a dictionary with the extra information, I think that is doable and mostly not breaking, as the iterations and belonging operations in Python are equally defined for lists and dictionary keys.
@memsharded,
Thank you for the explanation.
If I understand you correctly it is safe to do the builds with the first "filename". More precisely this is the correct way.
I do not miss the full freedom to choose from the available profiles, just wanted to be sure, how to use the merged build order file idiomatically.
Maybe it would be better to convert filenames to a dictionary with the extra information, I think that is doable and mostly not breaking, as the iterations and belonging operations in Python are equally defined for lists and dictionary keys.
May I respectfully remind you, that processing build order files are not part of the conan codebase (yet?) thus no assumptions could safely be made about the processing language, modes of parsing etc. A version field in the build order file might help avoid these kind of issues.
If I understand you correctly it is safe to do the builds with the first "filename". More precisely this is the correct way.
yes, exactly, this is the correct way, choosing other filename rather than the first one can be problematic as you have detected.
I do not miss the full freedom to choose from the available profiles, just wanted to be sure, how to use the merged build order file idiomatically.
Sounds good, thanks for the feedback. I was just having a look at the complexity of implementing this, doesn't seem very difficult, but not completely obvious either, so I might wait a bit until further users requests if they happen.
May I respectfully remind you, that processing build order files are not part of the conan codebase (yet?) thus no assumptions could safely be made about the processing language, modes of parsing etc. A version field in the build order file might help avoid these kind of issues.
Yes, that is true, good point. I was also thinking about it, and if done, it might be better to not modify the current field, but add with a new field, which is probably the easiest way to move forward without breaking and without versioning the file. Still, we will hold this until there are some other user specific requests.
I'll move this ticket to the docs to improve and clarify this behavior there. Thanks again!
Great, thank you!
Yes, that is true, good point. I was also thinking about it, and if done, it might be better to not modify the current field, but add with a new field, which is probably the easiest way to move forward without breaking and without versioning the file. Still, we will hold this until there are some other user specific requests.
Yes, this is the other non-breaking solution.
Doing https://github.com/conan-io/docs/pull/4116 PR to the docs for clarifying this.
Closed by documentation PR https://github.com/conan-io/docs/pull/4116, this docs will be in 2.18 release. Thanks again for your feedback!