protocolbuffers
[Git `main`] `-Dprotobuf_BUILD_LIBUPB:BOOL=OFF` is broken (and not covered by CI?)
Hi!
Compilation with -Dprotobuf_BUILD_LIBUPB:BOOL=OFF is currently broken on Git main (commit 24830cf07e559e912fa2622d3487a679ef0df9a9). For proof and/or to reproduce:
# cd "$(mktemp -d)"
# git clone --depth 1 https://github.com/protocolbuffers/protobuf
# cd protobuf/
# git rev-parse HEAD
24830cf07e559e912fa2622d3487a679ef0df9a9
# cmake -Dprotobuf_BUILD_LIBUPB:BOOL=OFF -S . -B build
# time make -C build VERBOSE=1
[..]
[ 43%] Building CXX object CMakeFiles/libprotoc.dir/upb_generator/common.cc.o
/usr/bin/c++ -DLIBPROTOC_EXPORTS -I/tmp/tmp.ajQg1hUGaR/protobuf/build -I/tmp/tmp.ajQg1hUGaR/protobuf -I/tmp/tmp.ajQg1hUGaR/protobuf/build/src -I/tmp/tmp.ajQg1hUGaR/protobuf/src -I/tmp/tmp.ajQg1hUGaR/protobuf/third_party/utf8_range -fvisibility=hidden -fvisibility-inlines-hidden -MD -MT CMakeFiles/libprotoc.dir/upb_generator/common.cc.o -MF CMakeFiles/libprotoc.dir/upb_generator/common.cc.o.d -o CMakeFiles/libprotoc.dir/upb_generator/common.cc.o -c /tmp/tmp.ajQg1hUGaR/protobuf/upb_generator/common.cc
In file included from /tmp/tmp.ajQg1hUGaR/protobuf/upb/reflection/common.h:15,
from /tmp/tmp.ajQg1hUGaR/protobuf/upb/reflection/def_pool.h:15,
from /tmp/tmp.ajQg1hUGaR/protobuf/upb/reflection/def.h:12,
from /tmp/tmp.ajQg1hUGaR/protobuf/upb/reflection/def.hpp:26,
from /tmp/tmp.ajQg1hUGaR/protobuf/upb_generator/common.h:16,
from /tmp/tmp.ajQg1hUGaR/protobuf/upb_generator/common.cc:8:
/tmp/tmp.ajQg1hUGaR/protobuf/upb/reflection/descriptor_bootstrap.h:14:10: fatal error: google/protobuf/descriptor.upb.h: No such file or directory
14 | #include "google/protobuf/descriptor.upb.h"
| ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
compilation terminated.
make[2]: *** [CMakeFiles/libprotoc.dir/build.make:1899: CMakeFiles/libprotoc.dir/upb_generator/common.cc.o] Error 1
make[2]: Leaving directory '/tmp/tmp.ajQg1hUGaR/protobuf/build'
make[1]: *** [CMakeFiles/Makefile2:186: CMakeFiles/libprotoc.dir/all] Error 2
make[1]: Leaving directory '/tmp/tmp.ajQg1hUGaR/protobuf/build'
make: *** [Makefile:136: all] Error 2
make: Leaving directory '/tmp/tmp.ajQg1hUGaR/protobuf/build'
This is on Linux with GCC 13 (and Protobuf 29.2 installed system-wide (if that matters)).
Best, Sebastian
CC @vitalybuka
PS: make -C build upb_generator/common.o could be of interest for the likely quickest way to see the same error.
We triage inactive PRs and issues in order to make it easier to find active work. If this issue should remain active or becomes active again, please add a comment.
This issue is labeled inactive because the last activity was over 90 days ago. This issue will be closed and archived after 14 additional days without activity.
![github-actions[bot] avatar](https://avatars.githubusercontent.com/in/15368?v=4 "Published by a pub.dev verified publisher"){kind=small}
This issue is about an actual bug in the build system and has three upvotes. Closing actual bugs as inactive and not replying to them at all for 7 months does not seem like a healthy project to me.
Sorry about the extended lack of response on this; we actually spend a lot of energy on triage (actively weekly triage meetings with 4+ sets of eyes, in addition to a staffed work-hours-only onduty rotation which fields tickets and questions, though we end up handling more internally than externally). Unfortunately even with that amount of effort, a lot of things still fall through the cracks for various reasons including github issues UI quirks. Unfortunately I don't think the active ping above to "@protocolbuffers" above will show up on anyones radar either.
In the future, we use the 'untriaged' label as the most useful tool for us to know that we need to look at it. We have configs and a bot which tries to automatically add it when issues are being bumped, but it appears to have not been set until this last reply in this case.
About the specific issue: the root cause of the issue here is that we have viewed upb as optional to build from the make build (as it was entirely separate from protoc), but recently the Rust code generator needs to use upb and so protoc has a upb dep here.
We are discussing internally what to do about this; in the most extreme case it may mean we just have to simply remove the Dprotobuf_BUILD_LIBUPB option, on the basis that upb has changed to become a non-optional dependency of protoc in that way.
Alternatively we maybe would have to simply disable the rust generator if this option is on (basically that --rust_out would need to error with a message that the binary was built without libupb and so cannot continue).
Is there a reason why you specially intended to turn this option off for your builds that we should be taking into account?
Thanks!
Thanks for the detailed reply.
For Buildroot this issue turned up when bumping versions (not applied yet). We generally try to keep build-times down by disabling features we don't need. In this case UPB is disabled for the host build: https://gitlab.com/buildroot.org/buildroot/-/blob/154f6694f641de1b9971ecd161644197526295d2/package/protobuf/protobuf.mk#L32
So enabling it is not a real showstopper but more a nuisance.
Hello @esrauchg, I appreciate your reply!
Let me reply to the concrete part first, and then segway into the maintenance topic:
About the specific issue: the root cause of the issue here is that we have viewed upb as optional to build from the make build (as it was entirely separate from protoc), but recently the Rust code generator needs to use upb and so protoc has a upb dep here.
We are discussing internally what to do about this; in the most extreme case it may mean we just have to simply remove the Dprotobuf_BUILD_LIBUPB option, on the basis that upb has changed to become a non-optional dependency of protoc in that way.
Alternatively we maybe would have to simply disable the rust generator if this option is on (basically that --rust_out would need to error with a message that the binary was built without libupb and so cannot continue).
Is there a reason why you specially intended to turn this option off for your builds that we should be taking into account?
I'm glad you ask! While helping https://github.com/google/libprotobuf-mutator upstream It became apparrent that Protobof has very long compile times (e.g. 62 minutes on a Lenovo ThinkPad X220 from 2012). I noticed that the build system spends time on compiling many things that libprotobuf-mutator would not need, e.g. it does not need protobuf-lite and no language support other than C++: So these features are compiled for nothing from the eyes of libprotobuf-mutator, its developers and its CI. I created a hack pull-request to stop compiling all that at https://github.com/google/libprotobuf-mutator/pull/267 there but it was rejected and rightfully so, because maintaining downstream patches for something that would ideally be fixed upstream for everyone and would need forward porting in the future is not great for maintainability. So I brought this up here via issue #20539 … and was ignored completely for four months. So the issue is about compile time, about multiplied waste of compute resources without clear need. I understand that someone with 2025 hardware will be affected less by this issue than someone with hardware from 2012… but then it also multiplies and why compile things you don't need.
Sorry about the extended lack of response on this; we actually spend a lot of energy on triage (actively weekly triage meetings with 4+ sets of eyes, in addition to a staffed work-hours-only onduty rotation which fields tickets and questions, though we end up handling more internally than externally). Unfortunately even with that amount of effort, a lot of things still fall through the cracks for various reasons including github issues UI quirks. Unfortunately I don't think the active ping above to "@protocolbuffers" above will show up on anyones radar either.
In the future, we use the 'untriaged' label as the most useful tool for us to know that we need to look at it. We have configs and a bot which tries to automatically add it when issues are being bumped, but it appears to have not been set until this last reply in this case.
Thanks for sharing that picture with me! I totally believe you but it looks very different from the outside. So much so that I was already starting to wonder how to best campaign publically to make "Google wake up and start properly maintaining Protobuf" for a while now, since March at least. The picture I see from the outside, with concrete examples:
- Issues that get no reply to for months like
- #20539,
- #20538 (right here), and
- #20373 .
- Issues that are being "discussed way" rather than improving the situation e.g.
- https://github.com/protocolbuffers/protobuf/issues/20591#issuecomment-3379596578 and
- https://github.com/protocolbuffers/protobuf/issues/20613#issuecomment-3379611902 .
- A list of open pull requests that does not look healthy (or maintained) at https://github.com/protocolbuffers/protobuf/pulls
- Major distros being stuck at 3.21.12 — both in Ubuntu 25.10 and Debian sid — while upstream is at v33 far far away. Superficially it looks like backwards-compatibility is not enough of a priority and that distros and upstream are two separate worlds by now that would need a reconnect. Distros probably need your help on multiple levels. Following SemVer is great in my book, but if distros need to have a package for v33, v32, v31 and so on in parallel, frequent breakage is likely not the answer (in any library).
It's hard to put into words how unhealthy and sad this situation looks to me. Too much of the ecosystem depends on Protobuf for it to not be in excellent shape. If you and your team can change that picture towards the better now, that would rock big time.
Have a nice day! :beers:
CC @hannob @joycebrum
A number of topics, happy to discuss them.
The specific status quo of the code is that protoc bin observable behavior truly does need the libupb. Its unfortunately not possible to support this option off and reproduce protoc binary that does what it does in general here. There's a related unfortunate quirk that we link most of our generators into protoc instead of having separate code generator plugins that would solve this shape of problem longer term but its a difficult move for us to switch everyone to.
Issues that get no reply to for months like https://github.com/protocolbuffers/protobuf/issues/20539,
Unfortunately this issue is in the same state as this one: it never got the untriaged tag on it, and similarly the ping you attempt was going into a void. Github and/or the bot have been broken in various ways that we have tied to catch, but this issue did fall through the cracks and was sitting in the void.
Generally we do expect nearly all users to use our released packages though, the cmake build from a random commit on main is not too common of a usecase. If you want to reopen that bug as untriaged we can take a look but cmake compile time is likely just pretty low on the totem pole for our investment as a less used case (which again is a scenario where we can take patches even if they are not otherwise something we can spend a lot of time investing on).
Issues that are being "discussed way" rather than improving the situation e.g.
https://github.com/protocolbuffers/protobuf/issues/20591#issuecomment-3379596578
https://github.com/protocolbuffers/protobuf/issues/20613#issuecomment-3379611902
Replied on that the thread to explain the state; the prior reply maybe wasn't clear that the status quo of that issue is understood to be fully-WAI, but we do accept PRs that would help someone else if they are harmless under expected configurations (that includes things not just warnings configs, unsupported compilers and OS, etc).
A list of open pull requests that does not look healthy (or maintained) at https://github.com/protocolbuffers/protobuf/pulls
Just in case its confusing and making it look worse than it is, nearly all PRs in the linked queue are created by copybara-service which the mechanism we have where Googlers can propose changes internally and they are automatically applied to the OSS project; those changes are reviewed internally, a PR is created just so we can run the CI, and if they end up applied internally they will be synced out separately. It's WAI part of the process that those PRs aren't actually for human review.
Major distros being stuck at 3.21.12 — both in Ubuntu 25.10 and Debian sid — while upstream is at v33 far far away.
Distros do have different needs than application developers, and broadly we do prioritize the latter over the former. It does make it so distros choose to live far behind the leading edge in this case; most of our users don't use the distro releases for their application development.
Note that v32 > v33 is not a breaking change an any language (we only ever do breaking change in any language in Q1 and we do quarterly minor releases). protoc is numbered only by minor since it goes with multiple different languages who all have different majors (and they all share the same minor so you can try to understand what they go with).
I think the distros chose to stick on 3.21 because C++Proto bumped to 4.22. It bumped at 5.26 and 6.30 and will bump again with 7.34 (C++ is also the only language that has bumped every year consistently, since we need to to deliver on performance critical changes for it). Broadly the supermajority of our C++Proto usage is on application development that doesn't use distro releases regardless and uses other package-manager releases (like brew) who are able to uptake our releases, or otherwise pick any arbitrary version and static-link it.
If there's feedback from distro maintainers we can discuss, but the status quo of distro's being behind while most users are on one of our supported releases (which are currently 5.x and 6.x, and will become 6.x and 7.x in 2026) isn't obviously a big problem (IIUC the distro owners are able to do security patches themselves in urgent situations if they need to).
A number of topics, happy to discuss them.
The specific status quo of the code is that protoc bin observable behavior truly does need the libupb. Its unfortunately not possible to support this option off and reproduce protoc binary that does what it does in general here. There's a related unfortunate quirk that we link most of our generators into protoc instead of having separate code generator plugins that would solve this shape of problem longer term but its a difficult move for us to switch everyone to.
@esrauchg would it be hard to have compile time switches for every language supported? They could be all on by default and nothing would break. I don't think it needs a plugin architecture to solve that problem, I see that as orthogonal.
Issues that get no reply to for months like #20539,
Unfortunately this issue is in the same state as this one: it never got the untriaged tag on it, and similarly the ping you attempt was going into a void. Github and/or the bot have been broken in various ways that we have tied to catch, but this issue did fall through the cracks and was sitting in the void.
I think that means that the void covered multiple months and all maintainers of the project. I find that worth noticing.
Generally we do expect nearly all users to use our released packages though, the cmake build from a random commit on main is not too common of a usecase.
I think it's fair to expect the latest main to work fine. The latest main is not a random commit to me.
If you want to reopen that bug as untriaged we can take a look but cmake compile time is likely just pretty low on the totem pole for our investment as a less used case (which again is a scenario where we can take patches even if they are not otherwise something we can spend a lot of time investing on).
I would have been open to create a pull request back in March provided a reply, buy-in to quick-to-reply review loop, and fewer pages of open pull requests.
Just in case its confusing and making it look worse than it is, nearly all PRs in the linked queue are created by
copybara-servicewhich the mechanism we have where Googlers can propose changes internally and they are automatically applied to the OSS project; those changes are reviewed internally, a PR is created just so we can run the CI, and if they end up applied internally they will be synced out separately. It's WAI part of the process that those PRs aren't actually for human review.
That's good to know. I think that info changes the picture in the sense that out of the three pages of pull requests then only the last page are stale but it also means that the pull request section is very alienating to non-Google contributors. It looks very weird.
PS: I'm not familiar with the term "WAI". Gemini suggests "Work Ability Index" but I fail to make sense of it.
Major distros being stuck at 3.21.12 — both in Ubuntu 25.10 and Debian sid — while upstream is at v33 far far away.
Distros do have different needs than application developers, and broadly we do prioritize the latter over the former. It does make it so distros choose to live far behind the leading edge in this case; most of our users don't use the distro releases for their application development.
I don't consider that a fair picture. Outside of LTS releases distros do have an interest to support the latest version of upstream software: being stuck in the past is not healthy, not in their interest. It's no trivial to make all packages work with the same version of Protobuf, and if you only have one central Protobuf, they all need to support that version at the same time: It becomes hard to move. That's one reason why backwards-compatibility is crucial.
If applications need post-distro versions, they will need to bundle Protobuf some way but in many ways bundling is not a solution — above all for security — so it's really in everyone's interest to have the latest protobuf ready in all non-LTS distros, as with Homebrew. I don't see how applications and distros should be even considered two different things, when applications are being compiled on distros. I'm not aware of any other distro package I have installed where a gap like that would be considered a feature or healthy.
If there's feedback from distro maintainers we can discuss, but the status quo of distro's being behind while most users are on one of our supported releases (which are currently 5.x and 6.x, and will become 6.x and 7.x in 2026) isn't obviously a big problem (IIUC the distro owners are able to do security patches themselves in urgent situations if they need to).
I think we need to distinguish between mainstream Ubuntu and Debian with 3.21.12 package and things like Gentoo and Arch with >v30 packages. My point is about Debian and Ubuntu here, about mainstream, about what is available in CI and most people's desktops. I would hope that Protobuf upstream and these two distros join a discussion about the future of Protobuf in these distros, to close that gap eventually.
I think really the legitimate high order bit here is still your >1 hour build concern; I reopened that issue and if you can update with your latest info on that we can advise. I don't believe that should be the case, and I don't believe that trying to trim libupb should really make any meaningful dent in your compile time.
I think that means that the void covered multiple months and all maintainers of the project. I find that worth noticing.
FWIW, over this period over 200 other issues were triaged. The void here was only the technical tooling failed to get your issues failed onto the review queues, our bug templates all put 'untriaged' tag upfront (so you may have used an unusual flow to open them; I tried and don't see how you can open an issue without untriaged tag on it upfront).
So its true that there's some holes in the triage process that you fell into, but I think the characteristic isn't quite as bad as you're imagining.
The latest main is not a random commit to me.
Our development model is that every individual change goes to main; essentially every commit we do is at one moment tip of main. We intend to keep the CI green against main, but it is the unstable development branch; it may be known for limited periods of time the behavior at tip-of-main is entirely arbitrary including potentially insecure / malformed serialization / etc.
We periodically cut release branches (like 32.x) which are what we do releases off, with tags for when the specific releases happen; if you want to build Protobuf from github to actually do something productive with, those are the only commits that we would recommend.
PS: I'm not familiar with the term "WAI". Gemini suggests "Work Ability Index" but I fail to make sense of it.
Sorry; "Working as Intended" (https://en.wiktionary.org/wiki/WAI)
It's no trivial to make all packages work with the same version of Protobuf, and if you only have one central Protobuf, they all need to support that version at the same time: It becomes hard to move
You did summarize the two sides of the coin of static vs dynamic linking summarized within one comment, and there fiery debate about which is better / more secure will continue past either of us being retired. Broadly speaking, essentially all of our users do static link (especially in e.g. Go I believe its not even possible to dynamically link at all). Obviously people will or will not, its only a 'meeting our users where they are, which is in static-linking' piece of context here.
I don't believe that trying to trim libupb should really make any meaningful dent in your compile time.
@esrauchg I see what you mean. It was just one of the things I played with disabling, and then reported that I found it broken, assuming that an existing flag was meant to be working.
You did summarize the two sides of the coin of static vs dynamic linking summarized within one comment, and there fiery debate about which is better / more secure will continue past either of us being retired. Broadly speaking, essentially all of our users do static link (especially in e.g. Go I believe its not even possible to dynamically link at all). Obviously people will or will not, its only a 'meeting our users where they are, which is in static-linking' piece of context here.
Golang can surely dynamically link to a C/C++ library, once you end up in CGO the language doesn't have much choice but to support both. It is of course true that golang source code is itself linked statically, because the golang language isn't quite so good at exposing a C ABI. It is also true that golang users self-select for people who don't like to link to the platform-provided C standard library -- but that is in no way an imposition of the language itself.
...
With regard to fiery debate, I don't think there is or should be suggested to be a a fiery debate in the first place. It isn't even a debate. Static linking is objectively useful for shipping standalone redistributable binaries, and shared linking is objectively useful for Linux distribution maintenance. No need to be fiery about it -- it is simply different use cases that have different needs. But both needs do exist, broadly speaking.
(If there's a fiery debate to be had, it will likely be about whether end users of software should feel more secure using standalone redistributable binaries or using Linux distributions. But that is neither here nor there.)
By the way I had no intentions to voice for static linking. Static linking to me is only acceptable if you rebuild and redeploy every time any of the direct or indirect dependencies change, and not miss any of these updates. That's challenging to get right in practice at best.
While the original discussions revolved around breakage on the main branch, currently this issue is present in v33.1. We disabled libupb for our build to strip out the unnecessary components, but resulted in build failures when updating to the latest release version.
While longer-term decisions are made in terms of internal usage of libupb are made, perhaps the CMake build should at least ensure it is included for other components that require it to avoid breaking downstream builds?
