bazelbuild
Bzlmod version could have better support for asking for a min/max range
Description of the feature request:
In cases like Protobuf-C++ and absl-cpp:
- Both projects are under active development and both make occasional small breaking changes, which have rolling deprecation windows
- Both don't really want two versions installed at a given time (generally can result in ODR violations when two versions are linked)
The status quo is that both projects just set module(compatibility_version=1) and don't change it; the current compatibility_version semantic doesn't really line up with this usecase because:
-
Because bazel uses the "maximum declared minimum-required version", the only thing bumping compatbility_version can really do is induce Bazel to want to install 2 versions, which would typically just be a breakage anyway
-
Most of the breaking changes actually only affect a small portion of users, so eg most people on protobuf-23 can use protobuf-29 even though there has officially been breaking changes in between (eg, removal or renaming of obscure and rarely used apis is a breaking change that actually breaks almost no one in practice)
-
The rolling compatibility window behavior means that the version compat numbers aren't really so 'clean', eg protobuf released in 2023 may realistically be compatible with absl from 2022-2024, protobuf released in 2022 may realistically be compatible with absl from 2021-2023.
The status quo ends up that protobuf and absl just declare compatibility_version=1 and never bump it. This has other undesirable behavior though where eg a binary uses:
- Library 1 declares that its min version is protobuf-23 + absl-20230101
- Library 2 declares its min version in absl-20240801
In reality the overall build will work with [protobuf-29 + absl-20240801] and maybe even work with [protobuf-23 + absl-20230101] (*this one actively contrary the claimed min version of library2), but bazel will select protobuf-23 + absl-20240801 which are actually incompatible with eachother and the build will not succeed with that combination of versions selected)
One possible way this could work better is if the protobuf module could specify a maximum absl version, and specify one that doesn't even exist at time of release (for example min=absl-20230101 max=absl-20240101). If the version number could be understood as an inequality then it would know that the version is expected to be forward-compatible with versions up to that release number. Then it could see that:
- Library 1 says >= protobuf-23, >= absl-20230101
- Library 2 says >= absl-2040801
- protobuf-23 says [absl >=20220101, absl <= 20240101]
- protobuf-27 says (some higher absl version)
And select protobuf-27 as the minimum version which does satisfies the constraints
Perhaps a similar capability could be work with maximum_compatiblity_version, though it would potentialy need some other adjustments to make it so absl + protobuf reasonably start using compatibility_version() with high versions (including, trying hard to preclude 2 versions being installed).
Which category does this issue belong to?
No response
What underlying problem are you trying to solve with this feature?
Reduce the amount of times that just taking a dep on a few libraries will result in bazel selecting a set of dependencies that are actually incompatible and don't build.
Which operating system are you running Bazel on?
No response
What is the output of bazel info release?
No response
If bazel info release returns development version or (@non-git), tell us how you built Bazel.
No response
What's the output of git remote get-url origin; git rev-parse HEAD ?
No response
Have you found anything relevant by searching the web?
No response
Any other information, logs, or outputs that you want to share?
No response
FYI @Wyverald @fmeum This is an interesting problem, I'm wondering if we can come up with a solution to declare a group of modules's interoperability without breaking the reproducibility of Bzlmod.
Is it possible to let module A (which doesn't depend on module B directly) declares if module B appears in the dependency graph, it must be a version >= X? This could potentially help abseil+protobuf when abseil introduces a breaking change that breaks protobuf.
Since we already have a mechanism for this (max_compatibility_level) that can also be used to make claims about compatibility with future versions (e.g. set it to a year), I think we should just use that.
If abseil includes a change that can conceivably break protobuf, then it will also likely break some other project. In that case, abseil should bump its compatibility level. This does cause churn for downstream projects and will ultimately make it less likely for them to update abseil, but I don't see a way around this.
Minimum version selection actually works well in this case since it will avoid updating abseil to the new version unless a dependency forces the update.
If we find this to be too painful in practice, we can make adjustments to this mechanism, but we have never really intentionally exercised it so far.
Since we already have a mechanism for this (max_compatibility_level)
I think the main problem is that if absl/protobuf start setting compatibility_level that it'll actually just necessarily break more people than are broken today.
The issue there is that bzlmod design follows the Go package manager design which presumes you can include two major versions to fix a diamond dependency problem, which is generally not true in C++ (it could be true if the C++ lib put major versions in their top level namespace, but its not idiomatic and at least Protobuf does not)
If you have a declared dependency graph that looks like: app --> lib1 --> protobuf-2019 -> lib2 --> protobuf-2021
We know there were some smaller breaking changes once per year. If these two protobuf libraries set compat_version 2019 and 2021 it will cause bazel to install 2 versions of protobuf in this case, and if lib1 and lib2 have cc_library that are linked into the same cc_binary that will be an ODR violation which could be a confusing linker error or even UB. That ends up much worse than the behavior we have today of it just installing protobuf-2021 which will probably work but may compile time break lib1 (in which case lib1 and lib2 very realistically are just impossible to use together in the same cc_binary, there's just really no solution that bazel can do to lead to this being a working binary).
I think to start using compatibility_version and max_compatibility_version we would need some behavior where we can tell Bazel to refuse to install two versions (= the behavior we get today by never bumping compatibility_version). Like maybe we can have a soft_compatibility_version where it should deterministically install only the highest min version in the case of multiple compatibility_versions in the chain and it should print a bug scary warning if there was any lower soft_compatibility_version which was violated due to that decision, telling the user eagerly that something may not work due to a conflicting diamond dependency problem and what are the two modules that have potentially incompatible deps.
If we had that behavior, then we could talk about protobuf + absl declaring their "range that we claim does work" guarantees in their module and then bzlmod could try to honor it where possible.
The issue there is that bzlmod design follows the Go package manager design which presumes you can include two major versions to fix a diamond dependency problem, which is generally not true in C++ (it could be true if the C++ lib put major versions in their top level namespace, but its not idiomatic and at least Protobuf does not)
It does follow the Go design, but not to the very end: If it finds requirements for two module versions with different compatibility levels, it just fails the build. Users can then opt into the Go behavior via multiple_version_override or accept that the new version may break them via single_version_override. You can't end up with two different versions of the same module in your dependency graph without an explicit override.
Have you tried this? This part of the code isn't exercised frequently today, so it's possible it doesn't work that way - but it should.
Yes, max_compatibility_level doesn't allow multiple compatibility levels of the same module to co-exist, it allows modules originally depending on a lower compatibility level to accept a higher compatibility level.
See this test (we should really better document this): https://github.com/bazelbuild/bazel/blob/c3677ba0fbbdb689b65c5c09980463ba7d920052/src/test/java/com/google/devtools/build/lib/bazel/bzlmod/SelectionTest.java#L402-L439
But I still don't know how this would help with the protobuf/absl situation exactly.
If absl doesn't increase its compatibility level, then obviously this solution doesn't work.
If absl increases its compatibility level for every breaking change, and protobuf sets max_compatibility_level for absl (assuming we somehow know this in advance), yes protobuf + absl will build or resolution will fail as needed. But other modules which depend on absl but not affected by the breaking change would either unnecessarily fail due to a different compatibility level was chosen or have to set max_compatibility_level at a proper level as well, which doesn't look an ideal solution either.
But other modules which depend on absl but not affected by the breaking change would either unnecessarily fail due to a different compatibility level was chosen or have to set max_compatibility_level at a proper level as well, which doesn't look an ideal solution either.
The first part is crucial here: Nobody can know whether other modules depending on absl are affected by a breaking change (or it wouldn't be breaking in the first place), so the prudent thing is to fail the build early and let the user opt into seeing potential breakages late (potentially even only at runtime, depending on the nature of the breaking change). Silently assuming that other modules remain compatible seems dangerous.
If a module is really certain that absl won't ever break them, they can always set max_compatibility_level to an artificially high value.
All of this is only relevant if absl starts bumping their compatibility level. If they only drop functionality that they are reasonably certain no reasonable module depends on, they probably shouldn't bump their compatibility level. But then they would also never break old protobuf versions.
Do we have examples of the line of breaking changes that absl has or is planning to make? Similar for protobuf. That could help us make a more informed decision.
Protobuf announced some of the breaking changes we intend to make in Q1 2025 here: https://protobuf.dev/news/2024-10-02/
Most of the breaking changes involve removal of deprecated APIs. Protobuf process intends to try to only make breaking changes once per year.
I'm not sure if absl has comms about planned upcoming breaking changes, but they do a good job documenting the type of breaking changes that they make in each of the past releases: https://github.com/abseil/abseil-cpp/releases
https://github.com/abseil/abseil-cpp/releases
I assume not every breaking change here will break protobuf? How often is protobuf broken and how do you track those breaking changes?
I think the most simple workaround might be to just specify both absl and protobuf explicitly in the root module's MODULE.bazel file and use --check_direct_dependencies=error to make sure the specified versions match the resolved versions. This way, you basically revert to the WORKSPACE behaviour where the root project controls exactly the versions fetched.
/cc @wffurr
https://github.com/bazelbuild/bazel/issues/25214 could probably help with this one
Since max versions are inherently incompatible with the MVS approach, I would go as far as suggesting that we should close this as a duplicate of #25214 and fold its use case into that issue.