libprotobuf complains about invalid UTF-8 data

[FaberFlorian]

Describe the bug The build aborts because non UTF-8-data seems to appear while building glibc-2.36.

To Reproduce Cannot paste my full cross-compile environment here. It happens when building glibc-2.36:

-----------------------arm target--------------------------

[..]
arm-linux-gnueabi-gcc --sysroot /home/target/sysroot  -nostdlib -nostartfiles -r -o /tmp/build_arm-linux-gnueabi/glibc-2.36_build/elf/librtld.os '-Wl,-(' /tmp/build_arm-linux-gnueabi/glibc-2.36_build/elf/dl-allobjs.os /tmp/build_arm-linux-gnueabi/glibc-2.36_build/elf/rtld-libc.a -lgcc '-Wl,-)' \
          -Wl,-Map,/tmp/build_arm-linux-gnueabi/glibc-2.36_build/elf/librtld.os.map
[libprotobuf ERROR /tmp/BearBuild/subprojects/Source/grpc_dependency/third_party/protobuf/src/google/protobuf/wire_format_lite.cc:618] String field 'rpc.Execution.EnvironmentEntry.key' contains invalid UTF-8 data when serializing a protocol buffer. Use the 'bytes' type if you intend to send raw bytes. 
[libprotobuf ERROR /tmp/BearBuild/subprojects/Source/grpc_dependency/third_party/protobuf/src/google/protobuf/wire_format_lite.cc:618] String field 'rpc.Execution.EnvironmentEntry.key' contains invalid UTF-8 data when parsing a protocol buffer. Use the 'bytes' type if you intend to send raw bytes. 
wrapper: failed with: gRPC call failed: 
make[7]: *** [Makefile:1336: /tmp/build_arm-linux-gnueabi/glibc-2.36_build/elf/librtld.os] Error 1```
-----------------------arm target--------------------------

-----------------------armhf target--------------------------
```arm-linux-gnueabihf-gcc --sysroot /home/target/sysroot  -shared -nostdlib -nostartfiles \
          -x assembler /dev/null -o /tmp/build_arm-linux-gnueabihf/glibc-2.36_build/format.lds.so
[libprotobuf ERROR /tmp/BearBuild/subprojects/Source/grpc_dependency/third_party/protobuf/src/google/protobuf/wire_format_lite.cc:618] String field 'rpc.Execution.EnvironmentEntry.key' contains invalid UTF-8 data when serializing a protocol buffer. Use the 'bytes' type if you intend to send raw bytes. 
[libprotobuf ERROR /tmp/BearBuild/subprojects/Source/grpc_dependency/third_party/protobuf/src/google/protobuf/wire_format_lite.cc:618] String field 'rpc.Execution.EnvironmentEntry.key' contains invalid UTF-8 data when parsing a protocol buffer. Use the 'bytes' type if you intend to send raw bytes. 
wrapper: failed with: gRPC call failed: 
make[6]: *** [Makerules:1033: /tmp/build_arm-linux-gnueabihf/glibc-2.36_build/format.lds] Error 1```
-----------------------armhf target--------------------------

**Environment:**
 - OS name: linux
 - OS version: L6.1.8-gentoo
 - OS architecture: x86_64
 - Bear version: bear 3.0.20
 - Bear install method: build from 3.0.21.tgz (yes, it identifies itself as 3.0.20!?)

**Additional context**
 - What build tools this project is using? binutils 2.40, gcc-11.3.1_p20230120
 - What architecture you are trying to compile for? arm-linux-gnueabihf, arm-linux-gnueabi 

[rizsotto]

Hey @FaberFlorian , thanks for the report.

This is an interesting problem. Could you run the build with bear --verbose -- ...? That might reveal what is the invalid UTF-8 the gRPC library is complaining about.

[FaberFlorian]

Yes, of course I tried a rerun with --verbose (have a nice 1.5GB log file here :) and the problem did not appear. Seems to be a heisenbug :)

[FaberFlorian]

I can try another set of runs in a few hours, for now I reverted to bear 2.x, it seems to work.

[rizsotto]

Yes, bear 2.x does not use gRPC. This bug won't be there. Maybe some others. :)

[nonenty]

Hello, @rizsotto

I encountered the same problem today. I tried to compile the kernel from android_kernel_hisi_hi3650. Bear has problem with this command

Debug log:

+ get_off drivers/huawei_platform/oases/inlinehook_offset.h drivers/huawei_platform/oases/inlinehook_func_pre.bin FUNC_PRE_PLACE_HOLDER_OFFSET 0xdead1001
+ _out=drivers/huawei_platform/oases/inlinehook_offset.h
+ _src=drivers/huawei_platform/oases/inlinehook_func_pre.bin
+ _key=FUNC_PRE_PLACE_HOLDER_OFFSET
+ _mem=0xdead1001
+ hex='\x01\x10\xad\xde'
++ echo -e '\x01\x10\xad\xde'
+ hex=$'\001\020\255\336'
++ LANG=C
++ LC_ALL=C
++ grep -obUaP $'\001\020\255\336' drivers/huawei_platform/oases/inlinehook_func_pre.bin
[libprotobuf ERROR google/protobuf/wire_format_lite.cc:577] String field 'rpc.Execution.arguments' contains invalid UTF-8 data when serializing a protocol buffer. Use the 'bytes' type if you intend to send raw bytes. 
wrapper: failed with: gRPC call failed: 

Simplified steps to reproduce:

1. Download inlinehook_subfunc_post.zip and extract it.
2. Run

bear -- grep -obUaP $'\005\020\255\336' out/drivers/huawei_platform/oases/inlinehook_subfunc_post.bin

3. Get the error output:

[libprotobuf ERROR google/protobuf/wire_format_lite.cc:577] String field 'rpc.Execution.arguments' contains invalid UTF-8 data when serializing a protocol buffer. Use the 'bytes' type if you intend to send raw bytes. 
[libprotobuf ERROR google/protobuf/wire_format_lite.cc:577] String field 'rpc.Execution.arguments' contains invalid UTF-8 data when parsing a protocol buffer. Use the 'bytes' type if you intend to send raw bytes. 
wrapper: failed with: gRPC call failed: 

Bear version 3.0.18, installed via apt on Ubuntu 22.04.

[FaberFlorian]

After having many more issues, I solved this by writing my own interceptor/CDB generator :)

Side effect is no slowdown of the build.

https://www.faberman.de/en/koala