AddressSanitizer: odr-violation on libupb_textformat_lib and libupb_json_lib

[zhtzeng]

What version of gRPC and what language are you using?

libgrpc++ 1.62.1/ C++

What operating system (Linux, Windows,...) and version?

Ubuntu 20.04

What runtime / compiler are you using (e.g. python version or version of gcc)

GCC14 with CMake

What did you do?

We use ASan to scan the generated binary.

What did you expect to see?

No ASan error

What did you see instead?

SUMMARY: AddressSanitizer: odr-violation: global 'google_protobuf_descriptor_proto_upb_file_layout' at /root/toolchain_build/google-cloud-cpp-2.22.0/grpc/src/core/ext/upb-gen/google/protobuf/descriptor.upb_minitable.c:1347:25 in libupb_textformat_lib.so.39

In https://github.com/grpc/grpc/blob/v1.57.x/CMakeLists.txt#L2487, we see libgrpc link a single libupb binary. After https://github.com/grpc/grpc/blob/v1.58.x/CMakeLists.txt#L2502, libupb is separated to multiple libraries and cause ODR violation. We also check the newer version and the same issue existed.

$ ldd libgrpc++.so.1.64.1 | grep upb
        libupb_json_lib.so.41 => /opt/3p/lib/libupb_json_lib.so.41 (0x00007f11311a0000)
        libupb_textformat_lib.so.41 => /opt/3p/lib/libupb_textformat_lib.so.41 (0x00007f113117b000)
        libupb_message_lib.so.41 => /opt/3p/lib/libupb_message_lib.so.41 (0x00007f113116c000)
        libupb_base_lib.so.41 => /opt/3p/lib/libupb_base_lib.so.41 (0x00007f1131169000)
        libupb_mem_lib.so.41 => /opt/3p/lib/libupb_mem_lib.so.41 (0x00007f1131165000)

Besides, libgrpc also has ODR violation.

ldd libgrpc.so.39.0.0 | grep upb
        libupb_textformat_lib.so.39 => /opt/3p/lib/libupb_textformat_lib.so.39 (0x00007fe62be43000)

Anything else we should know about your project / environment?

[drfloob]

@veblush PTAL, these are more your domains.

@zhtzeng What commands did you use to build gRPC? We have continuous builds using CMake, so I'm guessing that you're invoking it in a way we don't test (or with different tooling/versions).

[ns-osmolsky]

For us, gRPC is configured this way:

cmake				   \
    -DCMAKE_INSTALL_PREFIX=/opt/3p \
    -DCMAKE_BUILD_TYPE=Release \
    -DBUILD_SHARED_LIBS=yes \
    -DgRPC_INSTALL=ON \
    -DgRPC_BUILD_TESTS=OFF \
    -DgRPC_ABSL_PROVIDER=package \
    -DgRPC_CARES_PROVIDER=package \
    -DgRPC_PROTOBUF_PROVIDER=package \
    -DgRPC_RE2_PROVIDER=package \
    -DgRPC_SSL_PROVIDER=package \
    -DgRPC_ZLIB_PROVIDER=package \
    -S . -B cmake-out

Also note, this the ASan-instrumented build:

 CFLAGS="-fsanitize=address -O2 -fno-omit-frame-pointer"

[zhtzeng]

Thank you. @ns-osmolsky.

@veblush @drfloob BTW, we ever merged all upb libraries to 1 single library but this single upb library still has ODR violation with libgrpc. Since libgrpc also compile codes from upb. You can see below libgrpc, libupb_textformat and libupb_json compile same codes and link together. https://github.com/grpc/grpc/blob/v1.62.x/CMakeLists.txt#L1974. https://github.com/grpc/grpc/blob/v1.62.x/CMakeLists.txt#L3574

[ns-osmolsky]

BTW, I see that the problematic line (https://github.com/grpc/grpc/blob/v1.62.x/CMakeLists.txt#L3574) was deleted from master in this giant change: #36323

[zhtzeng]

BTW, I see that the problematic line (https://github.com/grpc/grpc/blob/v1.62.x/CMakeLists.txt#L3574) was deleted from master in this giant change: #36323

Remove that file is not enough. There are still many ODR violations. For example, libupb_textformat_lib and libupb_json_lib compile third_party/upb/upb/mini_descriptor/internal/base92.c https://github.com/grpc/grpc/blob/master/CMakeLists.txt#L3661 https://github.com/grpc/grpc/blob/master/CMakeLists.txt#L3861 ASan still complain ODR violation of _kUpb_FromBase92 in third_party/upb/upb/mini_descriptor/internal/base92.c. That is why I doubt v1.58 separate libupb to multiple upb libraries and easily cause ODR violations.

[veblush]

Yeah, something seems off around upb handling. We're going to replace bespoke upb build with the actual upb cmakefile from protobuf so this will be addressed eventually. I'll take a look at this if this can be resolved before this change.

[ns-osmolsky]

For the record, we deduped the lib specs and the issue went away.

[coryan]

@veblush I found a similar problem trying to use bzlmod. They may be completely different bugs, as this is related to CMake.