conan-io
[question] What's the best way to get version ranges select the minimum
What is your question?
I'm looking to see if there's any way to get Conan to select the minimum version from a range, and not the maximum? I work on a system with a relatively complex dependency graph, That is, to use NuGet style versioning.
Has anyone else achieved this, or have ideas on how it could be done?
Right now, I'm thinking of things like whether it is possible to hook into the function that resolves dependencies to an exact version in the graph, or to write an external tool that generates a lockfile to be passed into conan install
Have you read the CONTRIBUTING guide?
- [x] I've read the CONTRIBUTING guide
Hi @AshleighAdams
Thanks for the question.
There is no way in Conan to select the minimum version. Note that when defining a version range, this is done exactly to be able to bring new future, not yet released versions of dependencies.
So if you have a recipe that has a requires("mydep/[>=1.0 <2]") and what you want is the minimum always, why not doing just requires("mydep/1.0")?
It is important to note that Conan doesn't implement a full SAT-solver for finding the joint compatible version accross all version-ranges constraints, you can read more about it in https://docs.conan.io/2/knowledge/faq.html#getting-version-conflicts-even-when-using-version-ranges, but basically, that is an NP-hard problem, and given the high cost of evaluating each hypothesis (due to the dynamic nature of the dependency graph based on configuration), the problem is intractable in practice.
Yeah, I understand Conan provides no official way, I'm looking to go down the unsupported route, and I've cobbled together a seemingly working prototype, by reimplementing just a couple of Conan's internal functions. I need to do some more testing with diamond problems, and I'll provide the code in an update along with the results of my testing at some point.
Regarding the fixed versions, we don't want to use those because we want the packages to be upgradable by downstream consumers, so essentially just >=x, but if some other package has >=x+1, that version gets selected instead. Roots cannot downgrade versions, only upgrade. In the case you get a diamond, the highest between them is selected. Because you're only looking at the minimums, the search space isn't too much, and it becomes a relatively simple recursive problem.
Ok, thanks for the feedback.
Regarding the fixed versions, we don't want to use those because we want the packages to be upgradable by downstream consumers, so essentially just >=x, but if some other package has >=x+1, that version gets selected instead. Roots cannot downgrade versions, only upgrade. In the case you get a diamond, the highest between them is selected. Because you're only looking at the minimums, the search space isn't too much, and it becomes a relatively simple recursive problem.
I still think the road will be more bumpy than expected, and issues with diamonds, nested diamonds, multibranches, etc, will probably affect you in ugly ways.
You might also want to consider other alternatives to that versioning, for example, by using server side strategies with multi repositories and promotions as explained in https://docs.conan.io/2/ci_tutorial/tutorial.html. In that way you control the version that gets resolved by just not putting new versions in the server repo when you still don't want it. Then, when a new version is desired, then instead of some recipe forcing a higher minimum bound, the new version is made available in the server, so it fits in the ranges naturally as the latest.
You might also want to consider other alternatives to that versioning, for example, by using server side strategies with multi repositories and promotions as explained in https://docs.conan.io/2/ci_tutorial/tutorial.html. In that way you control the version that gets resolved by just not putting new versions in the server repo when you still don't want it.
That wouldn't work for us either, we don't directly control the consumers of the libraries here, so some of our developers will want a newer version, others not. Using channels or different remotes I think would've made things much more complex than simpler
So, we've been trialing it out, and honestly it's been refreshingly good for me. Less things breaking, a breaking change in an alpha could be safely broken and propitiated down the chain. The only issue we have is we need to manually resolve diamond problems with override=True, I plan to do some more investigation into this to see if this can automatically promote the largest value, but for now overriding is working well.
We did run into an issue where a diamond upgraded to an alpha, but the other side of the diamond was using an earlier semver core version alpha, and so did not accept the version bump as valid. So now we're forcing include_prereleases to True, as it's only needed for max-selecting version resolution strategies to begin with.
The dependency chain is quite complex too, so I have enough confidence to post the code now:
import re
import os
min_version_format_rx = re.compile(r"^>=\s*(?P<min_version>[A-Za-z0-9\.\-]+)(\+[A-Za-z0-9\.\-\+]*)?\s*$")
semver_rx = re.compile(r"^(?P<version>[A-Za-z0-9\.\-]+)(\+[A-Za-z0-9\.\-\+]*)?$")
def use_nuget_versioning(enable_debug_log: bool = False):
if NugetStyleRangeResolver.using_nuget_versioning:
return
NugetStyleRangeResolver.using_nuget_versioning = True
NugetStyleRangeResolver.enable_debug_log = enable_debug_log
NugetStyleRangeResolver.override_conan1()
class NugetStyleRangeResolver:
replaced = False
using_nuget_versioning = False
enable_debug_log = False
@staticmethod
def debug_log(msg):
if NugetStyleRangeResolver.enable_debug_log:
print(f"[nuget versioning] {msg}")
@staticmethod
def info_log(msg):
print(f"[nuget versioning] {msg}")
@staticmethod
def override_conan1():
# https://github.com/conan-io/conan/blob/1.66.0/conans/model/requires.py
# https://github.com/conan-io/conan/tree/1.66.0/conans/client/graph
# https://github.com/conan-io/conan/blob/1.66.0/conans/client/graph/graph_builder.py
# https://github.com/conan-io/conan/blob/1.66.0/conans/client/graph/range_resolver.py
if NugetStyleRangeResolver.replaced:
return
NugetStyleRangeResolver.replaced = True
from conans.errors import ConanException
from conans.client.graph.range_resolver import satisfying, RangeResolver, _parse_versionexpr
from conans.search.search import search_recipes
from semver import make_range, semver as make_semver, InvalidTypeIncluded, Range
def min_satisfying(versions, range_, loose=False, include_prerelease=False):
try:
range_ob = make_range(range_, loose=loose)
except InvalidTypeIncluded:
raise
except ValueError as e:
return None
min_ = None
min_sv = None
for v in versions:
if range_ob.test(v, include_prerelease=include_prerelease): # satisfies(v, range_, loose=loose)
sv = make_semver(v, loose=loose)
if min_ is None or sv.compare(min_sv) == -1: # compare(max, v, true)
min_ = v
min_sv = sv
return min_
def satisfying(list_versions, versionexpr:str, result):
from semver import SemVer, Range, max_satisfying
import re
original_versionexpr = versionexpr
select_max = False
if versionexpr.startswith("+"):
versionexpr = versionexpr[1:]
select_max = True
else:
# minimum version selection doesn't care about upper bounds, so force it to true
include_prerelease = True
# fix upper bounds including prereleases
versionexpr = re.sub(r"(\<)(\d(\.\d)*)(\s|,|$)", r"\1\2-\4", versionexpr)
version_range, loose, include_prerelease = _parse_versionexpr(versionexpr, result)
NugetStyleRangeResolver.debug_log(f"{original_versionexpr} => {versionexpr}")
# Check version range expression
try:
act_range = Range(version_range, loose)
except ValueError:
raise ConanException("version range expression '%s' is not valid" % version_range)
# Validate all versions
candidates = {}
for v in list_versions:
try:
NugetStyleRangeResolver.debug_log(f"\t{v}")
ver = SemVer(v, loose=loose)
candidates[ver] = v
except (ValueError, AttributeError):
result.append("WARN: Version '%s' is not semver, cannot be compared with a range"
% str(v))
# Search best matching version in range
result = None
if select_max:
result = max_satisfying(candidates, act_range, loose=loose, include_prerelease=include_prerelease)
else:
result = min_satisfying(candidates, act_range, loose=loose, include_prerelease=True)
NugetStyleRangeResolver.debug_log(f"={result}")
return candidates.get(result)
def _resolve_local(self, search_ref, version_range):
local_found = search_recipes(self._cache, search_ref)
local_found = \
[ref for ref in local_found
if ref.user == search_ref.user and
ref.channel == search_ref.channel]
if local_found:
ret = self._resolve_version(version_range, local_found)
range_match = min_version_format_rx.search(version_range)
if ret is not None and range_match is not None:
min_version = range_match.group("min_version")
version_match = semver_rx.search(ret.version)
ret_version = version_match.group("version")
is_online = os.getenv("INSEINC_OFFLINE") != "1"
if is_online and min_version != ret_version:
NugetStyleRangeResolver.info_log(f"{ret}: Expected to find version {min_version} via {version_range}, forcing update")
return None
return ret
og_conflicting_references = None
@staticmethod
def _conflicting_references(previous, new_ref, consumer_ref=None):
try:
if previous.ref.copy_clear_rev() != new_ref.copy_clear_rev():
if consumer_ref:
new_ref_semver = make_semver(new_ref.version, loose=True)
previous_semver = make_semver(previous.ref.version, loose=True)
latest_version = None
if new_ref_semver.compare(previous_semver) == -1:
latest_version = previous.ref
else:
latest_version = new_ref
return ("Conflict in %s:\n"
" %s\n required from %s\n"
" %s\n required from %s\n"
"To resolve the conflict, upgrade the dependency with:\n"
" self.requires(\"%s\", override=True)"
% (consumer_ref,
new_ref, consumer_ref,
previous.ref, next(iter(previous.dependants)).src,
latest_version))
return "Unresolvable conflict between {} and {}".format(previous.ref, new_ref)
except:
return og_conflicting_references(previous, new_ref, consumer_ref)
# now replace the functions
import importlib
graph_builder = importlib.import_module("conans.client.graph.graph_builder")
og_conflicting_references = graph_builder.DepsGraphBuilder._conflicting_references
graph_builder.DepsGraphBuilder._conflicting_references = _conflicting_references
range_resolver = importlib.import_module("conans.client.graph.range_resolver")
range_resolver.satisfying = satisfying
range_resolver.RangeResolver._resolve_local = _resolve_local
importlib.reload(importlib.import_module("conans.model.graph_lock")) # TODO: this?
importlib.reload(importlib.import_module("conans.client.graph.graph_builder")) # TODO: this?
We've put this in our Conan config directory, so it gets pulled in with conan config install, then there's a preamble script that will setup our imports (as we can't use python requires for this). use_nuget_versioning() is then invoked right at the start. All dependencies in our chain have had this preamble applied. External dependencies don't use version ranges to begin with as they originally always proved too unstable.
required_conan_version = ">=1.62.0"
from conans.paths import get_conan_user_home
exec(open(f"{get_conan_user_home()}/.conan/extensions/boot.py").read())
from extensions.versioning import use_nuget_versioning
use_nuget_versioning()
Here's what our dependency graph looks like in full:
@memsharded In case you don't see the value in this method of versioning, I found that vcpkg and go both use this method too, and there's some interesting docs that have been written on the subject: https://research.swtch.com/vgo-mvs
Perhaps it may change your opinion to make this a first-party opt-in feature, at least for Conan 2.0
For more context @memsharded this is the problem we encountered today:
Our recipes are using a version range requirement for OpenSSL so that consumers can control the exact version they use and update to the latest OpenSSL version for all those CVE updates, and this is only possible because of OpenSSL's very strict API/ABI version guarantees across major releases (https://docs.openssl.org/3.0/man7/migration_guide/#versioning-scheme).
In our equivalent of conan-center-index we provide OpenSSL 3.0.x and 3.5.x versions, the problem is that when we build libcurl which uses a version range requirement it is building against OpenSSL 3.5 and linking against a function which only was added in 3.4+, but the downstream consumers of our conan recipes were using the latest 3.0.x release where this function does not exist. And if you think about it more generally - it is a common pattern in C/C++ to optionally link against newer functions of libraries with preprocessor or build system based feature detection.
I accept that if you are resolving the whole graph where every conan recipe is built at the same time than the downstream consumer such that you can build and link every recipe against the exact resolved dependency version then sure the current approach of selecting the latest version is fine, but if you build all your conan recipes in CI and cache them in artifactory and separately your downstream consumers resolve their own graph and build their product independently of the conan recipes then you will end up with the problem as I encountered with libcurl. And given that is the model that conan-center-index operates on, and presumably so does the majority of people using conan without conan-center-index, this does seem like a fundamental issue with version ranges in conan and surely the packages built in conan-center-index are susceptible to this?
I am grateful that you shared your solution - I have not done any experiments with this yet but I might try a more manual solution where we manually force recipes in CI to build against specific dependency versions specified in the config.yml for a recipe - this might be sufficient for us as in general we also try to avoid using version ranges because they seem to constantly cause problems in the graph, but we do have a few libraries such as OpenSSL where using ranges seems unavoidable.
@bentonj-omnissa Happy to help, I've also updated the code to include changes we since made to resolve some issues we had here and there
Thanks for the feedback. I did a further thought about this some time ago. While it does look like manageable by the code shared above, that might be because it is a Conan 1 code. One of the potential issues in order to extrapolate this to Conan 2 is that in Conan 2 lockfiles rely on this assumption.
Locking dependencies is challenging, because there are dependency graphs that can contain different versions of the same package, or even different revisions of the same version of the same package. This is something that other package managers in C/C++ or in other languages are not capable of managing, but in Conan yes, you can depend both on zlib/1.2 and zlib/1.3, simultaneously, in the same dependency graph.
How can a lockfile handle that? Lockfiles in Conan 1 were a full graph representation to deal with it. But that made lockfiles a massively complicated, fragile, and capability-lacking feature in Conan 1. Conan2 did a great simplification, and now Conan 2 lockfiles are just a list of references, sorted by version and revision.
So they rely on the assumption that version ranges, similarly to recipe-revisions, naturally resolve to the newest available in the range. Lockfiles might be easily broken from a different version range resolution policy.
I still have interest in this, but at this point the interest is mostly academic, I am willing to give this a try to learn in the experience, but so far it still sounds very unlikely that this will be something that could be made built-in in Conan, the feature has interest, I am not denying it, but the cost/value ratio (weighted with the percentage of users that would want this feature) so far seems super-high. Let's see, I'll try to experiment with this when possible.