Build failes when imported in Bazel

[tinkerbeast]

What version of protobuf and what language are you using?

3.25.0 (https://github.com/protocolbuffers/protobuf/archive/v3.25.0.tar.gz)

What operating system (Linux, Windows, ...) and version?

Linux 6.2.0-36-generic 22.04.1-Ubuntu x86_64

What runtime / compiler are you using (e.g., python version or gcc version)

gcc (Ubuntu 11.4.0-1ubuntu1~22.04) 11.4.0

What did you do?

Using Bazel to build a project. The configuration is as follows:

http_archive(
    name = "com_google_protobuf",
    sha256 = "540200ef1bb101cf3f86f257f7947035313e4e485eea1f7eed9bc99dd0e2cb68",
    strip_prefix = "protobuf-3.25.0",
    urls = ["https://github.com/protocolbuffers/protobuf/archive/v3.25.0.tar.gz"],
)
load("@com_google_protobuf//:protobuf_deps.bzl", "protobuf_deps")
protobuf_deps()

What did you expect to see

Buiild go through.

What did you see instead?

/usr/bin/gcc -U_FORTIFY_SOURCE -fstack-protector -Wall -Wunused-but-set-parameter -Wno-free-nonheap-object -fno-omit-frame-pointer -g0 -O2 '-D_FORTIFY_SOURCE=1' -DNDEBUG -ffunction-sections -fdata-sections -MD -MF bazel-out/k8-opt-exec-2B5CBBC6/bin/external/upb/_objs/reflection_internal_stage1/file_def.d '-frandom-seed=bazel-out/k8-opt-exec-2B5CBBC6/bin/external/upb/_objs/reflection_internal_stage1/file_def.o' '-DBAZEL_CURRENT_REPOSITORY="upb"' -iquote external/upb -iquote bazel-out/k8-opt-exec-2B5CBBC6/bin/external/upb -iquote external/utf8_range -iquote bazel-out/k8-opt-exec-2B5CBBC6/bin/external/utf8_range -isystem external/upb/upb/reflection/stage1 -isystem bazel-out/k8-opt-exec-2B5CBBC6/bin/external/upb/upb/reflection/stage1 -g0 '-std=c99' -Wall -Wstrict-prototypes -fno-canonical-system-headers -Wno-builtin-macro-redefined '-D__DATE__="redacted"' '-D__TIMESTAMP__="redacted"' '-D__TIME__="redacted"' -c external/upb/upb/reflection/file_def.c -o bazel-out/k8-opt-exec-2B5CBBC6/bin/external/upb/_objs/reflection_internal_stage1/file_def.o)
# Configuration: cb91f35147ccf5b6d5bbb0ffdd28a9ceca503f7d25a8ae5b749c9e94b0133bfb
# Execution platform: @local_config_platform//:host

Use --sandbox_debug to see verbose messages from the sandbox and retain the sandbox build root for debugging
In file included from external/upb/upb/reflection/file_def.c:37:
external/upb/upb/reflection/file_def.c: In function '_upb_FileDef_Create':
external/upb/upb/port/def.inc:328:23: error: invalid initializer
  328 | #define UPB_DESC(sym) google_protobuf_##sym
      |                       ^~~~~~~~~~~~~~~~
external/upb/upb/reflection/file_def.c:246:28: note: in expansion of macro 'UPB_DESC'
  246 |   upb_StringView edition = UPB_DESC(FileDescriptorProto_edition)(file_proto);
      |                            ^~~~~~~~

Anything else we should know about your project / environment

No

[zhangskz]

@acozzette Is this a consequence of the monorepo merge?

[acozzette]

@tinkerbeast How are you getting a dependency on the @upb repo? We recently merged upb into the main protobuf repo, but the error messages make it appear like your project somehow still depends on the old upb repo.

[tinkerbeast]

I use the Bazel build which fetches any dependencies. I do not explicitly fetch the @upb repo. Here is my full workspace file -

load("@bazel_tools//tools/build_defs/repo:http.bzl", "http_archive")

http_archive(
    name = "com_google_protobuf",
    strip_prefix = "protobuf-3.25.0",
    urls = ["https://github.com/protocolbuffers/protobuf/archive/v3.25.0.tar.gz"],
)
load("@com_google_protobuf//:protobuf_deps.bzl", "protobuf_deps")
protobuf_deps()

http_archive(
    name = "com_github_grpc_grpc",
    strip_prefix = "grpc-1.59.2",
    urls = ["https://github.com/grpc/grpc/archive/v1.59.2.tar.gz"],
)
load("@com_github_grpc_grpc//bazel:grpc_deps.bzl", "grpc_deps")
grpc_deps()
load("@com_github_grpc_grpc//bazel:grpc_extra_deps.bzl", "grpc_extra_deps")
grpc_extra_deps()

[zhangskz]

I don't believe gRPC v1.59.2 is compatible with Protobuf 3.25.0 release yet, though we've already upgraded their main branch to be compatible with recent changes to merge the upb repo into the protobuf repo. I believe the next gRPC release should be compatible. In the meantime, you likely need to continue using 3.24.x with gRPC.

Please reopen if you still hit any issues, or if this is still broken in the next gRPC release.

[tinkerbeast]

I tried Protobuf v3.24.4 and v3.23.4 and they both give the same error -

ERROR: /tmp/build_output/a08c2e4811c846650b733c6fc815a920/external/upb/upbc/BUILD:64:28: Executing genrule @upb//upbc:gen_plugin_upb_proto_stage1 [for tool] failed: (Exit 1): process-wrapper failed: error executing command 
  (cd /tmp/build_output/a08c2e4811c846650b733c6fc815a920/sandbox/processwrapper-sandbox/12/execroot/__main__ && \
  exec env - \
    PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin \
    TMPDIR=/tmp \
  /tmp/build_output/install/117cee491f5c7d83be6e3c6d6b5b8ca4/process-wrapper '--timeout=0' '--kill_delay=15' '--stats=/tmp/build_output/a08c2e4811c846650b733c6fc815a920/sandbox/processwrapper-sandbox/12/stats.out' /bin/bash -c 'source external/bazel_tools/tools/genrule/genrule-setup.sh; bazel-out/k8-opt-exec-2B5CBBC6/bin/external/com_google_protobuf/protoc --plugin=protoc-gen-upb=bazel-out/k8-opt-exec-2B5CBBC6/bin/external/upb/upbc/protoc-gen-upb_stage0 -Isrc --upb_out=bazel-out/k8-opt-exec-2B5CBBC6/bin/external/upb/upbc/stage1 google/protobuf/compiler/plugin.proto')
src: warning: directory does not exist.
Could not make proto path relative: google/protobuf/compiler/plugin.proto: No such file or directory

I also tied Protobuf 3.25.0 with gRPC v1.58.2 and v1.57.1 and they both give the previous error.

Can you please tell me the last two compatible working versions?

[zhangskz]

gRPC v1.60.0 (prereleased) should be the first gRPC version compatible with Protobuf v25.0, due to gRPC dependency on protobuf's internal upb repo which was moved to the protobuf repo in v25.0. I believe gRPC v1.59.2 uses Protobuf v24.3 (see grpc) and should be compatible. Can you confirm you hit this still at compatible versions?

What command are you running when you hit this error?

I think the way oss_src_files is used here seems a bit off to me in https://github.com/protocolbuffers/protobuf/blob/70876bd87ea9050b46c8efdf191c246cf452349e/upb_generator/bootstrap_compiler.bzl#L95 since I think src_files should include the src/ prefix when used for https://github.com/protocolbuffers/protobuf/blob/70876bd87ea9050b46c8efdf191c246cf452349e/upb_generator/BUILD#L74

@haberman WDYT?

[tinkerbeast]

I tried with compatible versions gRPC v1.59.2 with Protobuf v24.3 and I still see the same error.

I changed up the directory structure a little since last time -

├── helloworld
│   ├── BUILD
│   ├── greeter_client.cc
│   ├── greeter_server.cc
│   └── helloworld.proto
└── WORKSPACE

The previous comment used a docker based Bazel to build. Bazel version is 6.4.0.

docker run \
  -e USER="$(id -u)" \
  -u="$(id -u)" \
  -v /home/rishin/workspace/grpc_getting_started:/src/workspace \
  -v /tmp/build_output:/tmp/build_output \
  -w /src/workspace \
  gcr.io/bazel-public/bazel:latest \
  --output_user_root=/tmp/build_output \
  build //helloworld:greeter_server

The error for this is -

ERROR: /tmp/build_output/a08c2e4811c846650b733c6fc815a920/external/upb/upbc/BUILD:68:28: Executing genrule @upb//upbc:gen_plugin_upb_proto_stage1 [for tool] failed: (Exit 1): bash failed: error executing command (from target @upb//upbc:gen_plugin_upb_proto_stage1) 
  (cd /tmp/build_output/a08c2e4811c846650b733c6fc815a920/sandbox/processwrapper-sandbox/12/execroot/__main__ && \
  exec env - \
    PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin \
  /bin/bash -c 'source external/bazel_tools/tools/genrule/genrule-setup.sh; bazel-out/k8-opt-exec-2B5CBBC6/bin/external/com_google_protobuf/protoc --plugin=protoc-gen-upb=bazel-out/k8-opt-exec-2B5CBBC6/bin/external/upb/upbc/protoc-gen-upb_stage0 -Isrc --upb_out=bazel-out/k8-opt-exec-2B5CBBC6/bin/external/upb/upbc/stage1 google/protobuf/compiler/plugin.proto')
# Configuration: cb91f35147ccf5b6d5bbb0ffdd28a9ceca503f7d25a8ae5b749c9e94b0133bfb
# Execution platform: @local_config_platform//:host

Use --sandbox_debug to see verbose messages from the sandbox and retain the sandbox build root for debugging
src: warning: directory does not exist.
Could not make proto path relative: google/protobuf/compiler/plugin.proto: No such file or directory

I also tried bazelisk-linux-amd64-v1_18_0, which in turn is using Bazel 6.4.0. It gives a similar error. However I also see sandboxing related errors. Let me know if you want me to post that here.

[tinkerbeast]

Any updates?

[zhangskz]

@haberman WDYT about https://github.com/protocolbuffers/protobuf/issues/14681#issuecomment-1812858673?

[zhangskz]

@tinkerbeast Can you check if this reproduces under Bazel 6.3?

[tinkerbeast]

I tried Bazel 6.3.2. It initially failed with a different error (about C++ version being at least 14). So I changed the build command to build --action_env=BAZEL_CXXOPTS="-std=c++17" - Which fails with the same error. Bazel versions 6.3.1 and 6.30 also yield the same error. I also tried C++ versions "c++14" and "c++2a", both of which fails with the same error.

[zhangskz]

@tinkerbeast Can you help provide a minimally reproducible example here (e.g. repo and command)? This would help us narrow down the prooblem here and if / how this could be related to your setup.

[juxuny]

gRPC v1.60.0 (prereleased) should be the first gRPC version compatible with Protobuf v25.0, due to gRPC dependency on protobuf's internal upb repo which was moved to the protobuf repo in v25.0. I believe gRPC v1.59.2 uses Protobuf v24.3 (see grpc) and should be compatible. Can you confirm you hit this still at compatible versions?

What command are you running when you hit this error?

I think the way oss_src_files is used here seems a bit off to me in

https://github.com/protocolbuffers/protobuf/blob/70876bd87ea9050b46c8efdf191c246cf452349e/upb_generator/bootstrap_compiler.bzl#L95

since I think src_files should include the src/ prefix when used for https://github.com/protocolbuffers/protobuf/blob/70876bd87ea9050b46c8efdf191c246cf452349e/upb_generator/BUILD#L74

@haberman WDYT?

I tried gRPC v1.59.2 and protobuf v24.3, the error is :

# Configuration: cb91f35147ccf5b6d5bbb0ffdd28a9ceca503f7d25a8ae5b749c9e94b0133bfb
# Execution platform: @local_config_platform//:host
ERROR: /home/yuanjie/.cache/bazel/_bazel_yuanjie/4402e32800daf0c7724281bc3b84bf6f/external/upb/upbc/BUILD:68:28: Executing genrule @upb//upbc:gen_plugin_upb_proto_stage1 [for tool] failed: (Exit 1): bash failed: error executing command (from target @upb//upbc:gen_plugin_upb_proto_stage1) /bin/bash -c ... (remaining 1 argument skipped)

Use --sandbox_debug to see verbose messages from the sandbox and retain the sandbox build root for debugging
src: warning: directory does not exist.
Could not make proto path relative: google/protobuf/compiler/plugin.proto: No such file or directory
ERROR: /home/yuanjie/.cache/bazel/_bazel_yuanjie/4402e32800daf0c7724281bc3b84bf6f/external/upb/BUILD:484:28: Executing genrule @upb//:gen_descriptor_upb_proto_stage1 [for tool] failed: (Exit 1): bash failed: error executing command (from target @upb//:gen_descriptor_upb_proto_stage1) /bin/bash -c ... (remaining 1 argument skipped)

Use --sandbox_debug to see verbose messages from the sandbox and retain the sandbox build root for debugging
src: warning: directory does not exist.
Could not make proto path relative: google/protobuf/descriptor.proto: No such file or directory
Target //cuda:cuda_shared failed to build
Use --verbose_failures to see the command lines of failed build steps.
INFO: Elapsed time: 44.786s, Critical Path: 35.09s
INFO: 1744 processes: 381 internal, 1361 linux-sandbox, 2 local.
FAILED: Build did NOT complete successfully

[tinkerbeast]

@zhangskz Sorry for the delayed response. The attached file contains a minimal working example. Environment is Ubuntu 22.04 with no protoc or grpc related libraries installed. Compilation with docker as before.

example.tar.gz

[jkaye2012]

We are experiencing this issue as well. I think it's not possible to use Protobuf with gRPC at all on versions of clang >= 15 because of this issue. Using older Protobuf versions causes builds to fail (because upb sets -Werror in its cc_library), and newer versions cannot be relied upon because of this incompatibility.

[haberman]

I think I have a fix in https://github.com/protocolbuffers/protobuf/pull/15829. This fix is backported from main.

I've verified in a local test that this addresses the Could not make proto path relative errors.

[zhangskz]

It looks like that fix should already be released in Protobuf v25.0+. 24.x does not have the fix, but is also out-of-support as users should upgrade to 25.x instead.

Most of these reports seem to be from 24.3, or 25.x with an incompatible gRPC version. Is this issue reproducible when using gRPC >= v1.60.0 with Protobuf >= 25.0 which should both include the fix and be compatible?

[jkaye2012]

Are there current releases that would allow us to test such a configuration? I tried upgrading our internal infrastructure to all permutations of releases for gRPC and protobuf (including the most recent commit of both) and still saw the issues present here. If you have an example of two specific commits that I should test against for protobuf and rules_proto_grpc, I can run the test again with those versions and report back.

If you do have such versions to try, do you also have a suggestion of clang version? Ideally we would be able to use 17 or 18, but I'm not sure if there are compatibility limits for the libraries at this time.

[github-actions[bot]]

We triage inactive PRs and issues in order to make it easier to find active work. If this issue should remain active or becomes active again, please add a comment.

This issue is labeled inactive because the last activity was over 90 days ago.

[github-actions[bot]]

We triage inactive PRs and issues in order to make it easier to find active work. If this issue should remain active or becomes active again, please reopen it.

This issue was closed and archived because there has been no new activity in the 14 days since the inactive label was added.