conan-io
[regression] validate() failures no longer fall back on compatibility() options.
What is your question?
As we port more cross-compile scenarios to conan 2, I just hit some cases where the compatibility() fallbacks of packages don't seem to work in conan2 anymore.
The basic scenario is that the packages have both a validate() function and a compatibility() function, that are intended to work together - validate flags scenarios that are not permissible, and then compatibility() provides a fallback to some version of the tool that should have been used.
First, to simplify things all the way to down to just a minimized test case, consider
class ToolConan(ConanFile):
name="tool"
version="1.0"
package_type = "application"
options = { "valid": [None, True] }
def validate(self):
if not self.options.valid:
raise ConanInvalidConfiguration("not valid")
def compatibility(self):
return [{"options": [("valid", True)]}]
def package_id(self):
self.output.info("package_id()")
Then do conan export-pkg tool/conanfile.py -o valid=True
and have a simple consumer
[tool_requires]
tool/1.0
do conan install consumer/conanfile.txt (which will default to valid=None)
Conan1 behaves as desired:
tool/1.0: Main binary package 'INVALID' missing. Using compatible package 'ac5c71e7fd9ce7e77912d3fbde55583f9ae8c6fa'
But conan2 stops after validate(), and never called compatibility()
Build requirements tool/1.0#dc79dea61f133b79036a95cba88113cd:da39a3ee5e6b4b0d3255bfef95601890afd80709 > - Invalid ERROR: There are invalid packages: tool/1.0: Invalid: not valid
Obviously in the real case(s), the condition was more complex than an option valid=True, but some motivating examples:
-
If the invalid case is a binary which cannot even exist (for whatever reason), it was handy to have
validate()explicitly say so - this prevents a mistaken attempt at--build missing, and also avoids conan wasting time pinging each of its artifactory remotes to see if they have it. By using validate() this way, it would know the primary binary was forbidden and proceed directly to the compatible options (at least in conan1). @memsharded and I had some discussion about this "offline usage" scenario years ago back in https://github.com/conan-io/conan/issues/9527#issuecomment-938287358 and https://github.com/conan-io/conan/issues/8074#issuecomment-846632671 and I think he actually suggested using validate() this fashion to me as a solution, but I haven't been able to a specific reply... maybe I found that workaround myself. In any case it's been a useful thing to do. -
There could be a tool package which is sensitive to compiler.version or cppstd (needs to be built with recent tools), but is fine with being called by consumers using an older versions (since they're just calling a pre-build executable). If you only have one binary package of the tool, you might handle this case by deleting the versions from its package_id completely, but if you maintain a couple builds want to pick the "closest without going over" kind of match, it was handy to return it as a list of ordered compatibility() options to try. Now if check_min_cppstd or something errors out, you're just done;
compatibility()isn't reached. -
We have a compiler-like tool (actually a binding generator, but its output depends on the target compiler/pointer size/etc), that can't quite get a fully-working
-m32/-m64kind of option because windows APIs likeGetRefTypeInfoinherently follow the native-architecture registry. So this wants to say that if settings_target (i.e. profile:host) is 32-bit, it wants to direct you to run the 32-bit version of the tool so those lookups find results matching profile:host. So it has something like:def _WoW64_target_arch(self): settings_target = getattr(self, 'settings_target', None) if settings_target and self.settings.os == 'Windows': def is_win64(arch): return str(arch) in ["x86_64", "ia64", "ppc64le", "ppc64", "armv8", "mips64"] if is_win64(self.settings.arch) and not is_win64(self.settings_target.arch): return self.settings_target.arch # a win64 process cannot opt into the WoW64 filesystem redirector, # (there's no inverse of Wow64DisableWow64FsRedirection). def validate(self): if WoW64_arch := self._WoW64_target_arch(): raise ConanInvalidConfiguration("Win64 (arch={arch_build}) tlimport cannot use WoW64 redirection to load typelibs for {arch} target. Please use --settings:build tlimport/*:arch={arch}".format(arch=WoW64_arch, arch_build=self.settings.arch)) def compatibility(self): result = [] if WoW64_arch := self._WoW64_target_arch(): result.append({"settings": [("arch", str(WoW64_arch))]}) return resultYou can't do this kind of thing in package_id() at all, at least as far as I can tell, because access to self.settings is forbidden, and self.settings_target isn't defined yet. And it seems like a bad idea anyway, because that would make it easy to end up creating mis-labeled binaries (you'd be mixing the producer/consumer roles), lying about what id this is, instead of explicitly proposing that one should use something else.
Have you read the CONTRIBUTING guide?
- [ ] I've read the CONTRIBUTING guide
@thorntonryan found another clue: this seems related to #17627 and #17637. If I do conan install --build compatible on my minimized test case then it does succeed, and I get
Requested binary package 'da39a3ee5e6b4b0d3255bfef95601890afd80709' invalid, can't be built Checking 1 configurations, to build a compatible one, as requested by '--build=compatible' Found compatible package '4d312be773bb045740bcbbca2ac85420aae4c898': [options], valid=True ... tool/1.0: Building from source tool/1.0: Package tool/1.0:4d312be773bb045740bcbbca2ac85420aae4c898
But that rebuilds tool/1.0 from source, every time. I haven't yet figured out a way to get conan 2 to use an existing compatible binary when the current configuration is invalid (as conan 1.0 did...)
Hi @puetzk
Thanks for the report.
This wouldn't be a regression, but Conan 2 provide a better distinction between validate() and validate_build().
See the following test:
def test_compatible_validate_error():
c = TestClient()
tool = textwrap.dedent("""
from conan import ConanFile
from conan.errors import ConanInvalidConfiguration
class ToolConan(ConanFile):
name="tool"
version="1.0"
package_type = "application"
options = { "valid": [None, True] }
def validate(self):
if not self.options.valid:
raise ConanInvalidConfiguration("not valid")
def compatibility(self):
return [{"options": [("valid", True)]}]
def package_id(self):
self.output.info("package_id()")
""")
c.save({"tool/conanfile.py": tool,
"app/conanfile.txt": "[tool_requires]\ntool/1.0"})
c.run("export-pkg tool/conanfile.py -o valid=True")
c.run("install app", assert_error=True)
assert "tool/1.0: Invalid: not valid" in c.out
# LETS USE validate_build() INSTEAD
c.save({"tool/conanfile.py": tool.replace("validate(", "validate_build(")})
c.run("export-pkg tool/conanfile.py -o valid=True")
c.run("install app")
assert "tool/1.0: Found compatible package" in c.out
This distinction is what allows to define a logic like:
def validate(self):
check_min_cppstd(self, 14)
def validate_build(self):
check_min_cppstd(self, 17)
Which allows a package to be built only with c++17 as minimum (c++17 only in internal implementation .cpp, not in API) but to be consumed with at least c++14 (c++14 used in the public headers of the library), a common scenario.
I'm aware of the validate/validate_build distinction, but I don't think they cover everything. They are documented as
The validate() method can be used to mark a package binary as “invalid”, or not working for the current configuration The validate_build() method is used to verify if a package binary can be built with the current configuration
So moving the check into validate_build() only seems like it would help if the difference comes into play when building, doesn't influence package_id, and doesn't matter when the binaries already exist . Your check_min_cppstd is a nice clean solution for the common case of a library whose public API is more permissive than its implementation, though. You can just have package_id(self): del self.info.settings.compiler.cppstd and everything can use the same package_id, with validate() more permissive than validate_build().
But that wouldn't be enough for trickier cases, e.g. a library that has #ifdef __cpp_* >= 20xx where c++14 and c++20 builds produce ODR-different binaries, and we want c++17 to take a compatibility() fallback to the c++14 binaries, and c++23/26 could use the either the c++20 or c++14 ones. As far as I know that kind of ranked "best match" search order can only happen via compatibility -- although it probably doesn't hurt (besides wasted space/time) to just have redundant c++17/23/26 binaries that are duplicates of the c++14/20 ones. And I suppose you could do something in package_id similar to https://docs.conan.io/2/reference/conanfile/methods/package_id.html#partial-information-erasure if you wanted to make sure you only end up exporting cppstd=14 and cppstd=20 versions. So you could probably avoid using compatibility() here.
But for case 3 example, you can build both arch=x86 and arch=x86_64 versions of the tool fine, you of course get separate package_ids containing different executables, and these are probably already in the cache (or at least on Artifactory). You just can't
use a 64-bit tool executable to process files from a 32-bit target.
Using validate_build() simply doesn't cover that, because conan doesn't care if validate_build() failes when a binary for the current package_id already exists. It worked when you pached my valid=None example there was no binary in the cache with valid=None (nor could there be, because validate_build wouldn't let you create it). But if we change to an example more like this scenario:
from conan import ConanFile
from conan.errors import ConanInvalidConfiguration
class ToolConan(ConanFile):
name="tool"
version="1.0"
package_type = "application"
settings = "arch"
options = { "valid": [None, True] }
def validate_build(self):
if self.settings.arch == 'x86_64' and getattr(self.settings_target, "arch", None)== 'x86':
self.output.error("64->32 cross compiling doesn't work")
raise ConanInvalidConfiguration("64->32 cross compiling doesn't work")
def compatibility(self):
self.output.info("compatibility()")
return [{"settings": [("arch", self.settings_target.arch)]}]
conan export-pkg tool/conanfile.py -s:h arch=x86 -s:b arch=x86
conan export-pkg tool/conanfile.py -s:h arch=x86_64 -s:b arch=x86_64
Then it doesn't work in conan 1.x or 2.x. conan install consumer -s:h arch=x86 -s:b arch=x86_64
does call validate_build(), I see the message from it get printed:
tool/1.0: ERROR: 64->32 cross compiling doesn't work
But they both ignore it, continue on, and pick the arch=x86_64 package_id from the cache anyway.
Which I think is reasonable and correct behavior given the documentation of validate_build() - they have an existing binary and don't need to do any building. They could even have skipped calling it at all. They don't, but in any case the result doesn't matter.
But that means just replacing validate() with validate_build() isn't a solution if you're trying to answer "does it work", rather than "can it be built".
Thanks for the extensive feedback.
But they both ignore it, continue on, and pick the arch=x86_64 package_id from the cache anyway.
But this is expected, indeed the validate_build() method is executed (it always is, even if not always necessary), but the result is discarded if there is an existing binary for it.
But that means just replacing validate() with validate_build() isn't a solution if you're trying to answer "does it work", rather than "can it be built".
That is true, the solution is not just a replacement, but for many cases it is actually a combination. Both validate_build() and validate() seems to be necessary in that case, if there are usage limitations, and we want the:
conan install consumer -s:h arch=x86 -s:b arch=x86_64
command to also raise an invalid error, then the validate() method is necessary too
In a test it would be like:
def test_compatible_validate_error():
c = TestClient()
tool = textwrap.dedent("""
from conan import ConanFile
from conan.errors import ConanInvalidConfiguration
class ToolConan(ConanFile):
name="tool"
version="1.0"
package_type = "application"
settings = "arch"
options = { "valid": [None, True] }
def validate_build(self):
if self.settings.arch == 'x86_64' and getattr(self.settings_target, "arch", None) == 'x86':
self.output.error("64->32 cross compiling doesn't work")
raise ConanInvalidConfiguration("64->32 cross compiling doesn't work")
def validate(self):
if self.settings.arch == 'x86_64' and getattr(self.settings_target, "arch", None) == 'x86':
self.output.error("64->32 cross compiling doesn't work")
raise ConanInvalidConfiguration("64->32 cross compiling doesn't work")
def compatibility(self):
self.output.info("compatibility()")
return [{"settings": [("arch", self.settings_target.arch)]}]
""")
c.save({"tool/conanfile.py": tool,
"app/conanfile.txt": "[tool_requires]\ntool/1.0"})
# c.run("export-pkg tool/conanfile.py -s:h arch=x86 -s:b arch=x86")
c.run("export-pkg tool/conanfile.py -s:h arch=x86_64 -s:b arch=x86_64")
c.run("install app -s:h arch=x86 -s:b arch=x86_64", assert_error=True)
assert "tool/1.0: Invalid: 64->32 cross compiling doesn't work" in c.out
c.run("install app -s:h arch=x86_64 -s:b arch=x86")
# It works, it doesn't fail
assert "tool/1.0: Found compatible package '62e589af96a19807968167026d906e63ed4de1f5': arch=x86_64" in c.out
Duplicating the check in validate_build() doesn't seem to change the behavior: conan install still fails if validate() rejects the current combination of settings and settings_target, and it still doesn't seem to try any of the suggested substitutions -- or even call compatibility() at all.
So far the only thing that works in conan 2 (skips the invalid configuration and tries the substitution returned from compatibility()) is --build compatible. Using that, it will call compatibility() and substitute the returned settings like conan1 did.
tool/1.0: Requested binary package '62e589af96a19807968167026d906e63ed4de1f5' invalid, can't be built tool/1.0: Checking 1 configurations, to build a compatible one, as requested by '--build=compatible' tool/1.0: Found compatible package '1b37be562194b3529b11722d4aedbef8c6adc28f': Build requirements tool/1.0#2434383ebec9f0a8ecbaf714a77eb11f:1b37be562194b3529b11722d4aedbef8c6adc28f - Build
Then it will (re-)build the x86 package (unnecessary since it already existed, but expected since there was a --build option)
======== Installing packages ========
-------- Installing package tool/1.0@re41236/testing (1 of 1) -------- tool/1.0@re41236/testing: Building from source tool/1.0@re41236/testing: Package tool/1.0@re41236/testing:1b37be562194b3529b11722d4aedbef8c6adc28f tool/1.0@re41236/testing: settings: arch=x86 tool/1.0@re41236/testing: options: valid=None tool/1.0@re41236/testing: compatibility_delta: settings=[('arch', <conan.internal.model.settings.SettingsItem object at 0x00000271F3C1F410>)]
and overall the conan install succeeds and picks the arch=x86 version of tool (that it just built).
Minor glitch here: I see that it showed up as conan.internal.model.settings.SettingsItem. That's my fault by having def compatibility(self): return [("arch", self.settings_target.arch)]}]. But maybe conan should verify the types that were returned, or else just call str(...) itself, because leaking the conan.internal.model.settings.SettingsItem like that seems like an easy mistake. Or maybe that's just an erroneous recipe :-)
In any case, I've yet to find any way to get conan2 to use an existing x86 package. --build compatible will rebuild every time, accumulating more and more package revisions... and without that it doesn't call compatibility() at anymore (if validate fails).
For starters, thanks for taking the time to respond in such depth to our question.
We're still getting the hang of Conan2's new model and trying to learn the ins and outs of how it affects our usage.
To clarify, perhaps, given your test example:
c.run("install app -s:h arch=x86 -s:b arch=x86_64", assert_error=True)
+ c.run("install app -s:h arch=x86 -s:b arch=x86_64")
Our particular goal happens to be the opposite behavior from what the test shows. This is the statement we expect to succeed and fallback to a compatible package.
The gory details about WoW64 behavior are described in greater detail in the "Expand" above. But the cross-compile direction here matters.
To try and restate what we're seeing:
-
Given a
validate()andcompatibility()method likeMore details
[tool_requires] tool/1.0conanfile.txt
from conan import ConanFile from conan.errors import ConanInvalidConfiguration from conan.tools.cmake import cmake_layout class ToolConan(ConanFile): name="tool" version="1.0" package_type = "application" settings = "arch", "os", #"build_type" def _WoW64_target_arch(self): settings_target = getattr(self, 'settings_target', None) if settings_target and self.settings.os == 'Windows': def is_win64(arch): return str(arch) in ["x86_64", "ia64", "ppc64le", "ppc64", "armv8", "mips64"] if is_win64(self.settings.arch) and not is_win64(self.settings_target.arch): return str(self.settings_target.arch) # a win64 process cannot opt into the WoW64 filesystem redirector, # (there's no inverse of Wow64DisableWow64FsRedirection). return None def validate(self): WoW64_arch = self._WoW64_target_arch() if WoW64_arch: raise ConanInvalidConfiguration("Win64 (arch={arch_build}) tool cannot use WoW64 redirection to load typelibs for {arch} target. Please use --settings:build tool:arch={arch}".format(arch=WoW64_arch, arch_build=self.settings.arch)) def compatibility(self): result = [] if WoW64_arch := self._WoW64_target_arch(): self.output.info(f"{WoW64_arch = }") result.append({"settings": [("arch", str(WoW64_arch))]}) return resulttool/conanfile.py
-
Export x86 tool
$ conan export-pkg tool/conanfile.py --profile:build x86 --profile:host x86This works. The x86 tool has been created and is in the cache -
Try to use it in a cross compile
$ conan install conanfile.txt --profile:build x86_64 --profile:host x86Installing it fails to find the compatible packageBuild requirements tool/1.0#e953af330fc13bc5f02566a10f907fca:522dcea5982a3f8a5b624c16477e47195da2f84f - Invalid ERROR: There are invalid packages: tool/1.0: Invalid: Win64 (arch=x86_64) -
Try to build it in a cross compile
$ conan install conanfile.txt --profile:build x86_64 --profile:host x86 --build=compatibleMore surprisingly, this succeeds?======== Computing necessary packages ======== tool/1.0: WoW64_arch = 'x86' tool/1.0: Requested binary package '522dcea5982a3f8a5b624c16477e47195da2f84f' invalid, can't be built tool/1.0: Checking 1 configurations, to build a compatible one, as requested by '--build=compatible' tool/1.0: Found compatible package 'c11e463c49652ba9c5adc62573ee49f966bd8417'
In any case, I've yet to find any way to get conan2 to use an existing x86 package. --build compatible will rebuild every time, accumulating more and more package revisions... and without that it doesn't call compatibility() at anymore (if validate fails).
So yeah. Conan can find the compatible package, but only when given build=compatible, and can't seemingly re-use what's already been built. In this way, it's just not behaving the way other build=missing type packages ordinarily behave.
