Crash with code 37 on envoy's remote compilation attempt

[amdei]

Bazelisk crashes with exit code 37 on attempt to run remote compilation of envoy proxy server.

Platform: Linux Debian 11, amd64.

Bazelisk installed as bazel:

# bazel version
Bazelisk version: v1.15.0
Starting local Bazel server and connecting to it...
Build label: 6.0.0
Build target: bazel-out/k8-opt/bin/src/main/java/com/google/devtools/build/lib/bazel/BazelServer_deploy.jar
Build time: Mon Dec 19 15:52:35 2022 (1671465155)
Build timestamp: 1671465155
Build timestamp as int: 1671465155

Both local and remote toolchains configured and working - envoy can be built successfully on both machines. This step is rather complicated - but I can provide details if needed.

Remote compilation with Buildfarm is set and ruining smoothly as per "Hello world!" quick start example:

# bazel clean
INFO: Starting clean (this may take a while). Consider using --async if the clean takes more than several minutes.
# bazel run --remote_executor=grpc://ng-btn-envoy-re.cp.ru:8980 :main
INFO: Invocation ID: 87fa5b48-af98-42e9-9c08-758c104380c1
INFO: Analyzed target //:main (36 packages loaded, 155 targets configured).
INFO: Found 1 target...
Target //:main up-to-date:
  bazel-bin/main
INFO: Elapsed time: 1.392s, Critical Path: 0.19s
INFO: 6 processes: 2 remote cache hit, 4 internal.
INFO: Build completed successfully, 6 total actions
INFO: Running command line: bazel-bin/main
Hello, World!

Observed behavior: But once I try to run compilation of envoy, it silently dies with exit code 37 (Unhandled Exception / Internal Bazel Error):

# time bazel build -c opt envoy --config=remote-clang-libc++ --remote_executor=grpc://ng-btn-envoy-re.cp.ru:8980 --verbose_failures --remote_local_fallback=false --jobs 64

real    0m0.703s
user    0m0.009s
sys     0m0.025s
# echo $?
37

Workaround (found by accident): add --profile flag. With this flag buld can be finished successfully:

# time bazel build -c opt envoy --profile --config=remote-clang-libc++ --remote_executor=grpc://ng-btn-envoy-re.cp.ru:8980 --verbose_failures --remote_local_fallback=false --jobs 64
Starting local Bazel server and connecting to it...
INFO: Invocation ID: e2bc26e7-09a7-4d21-94e0-a6739012815c
DEBUG: /root/.cache/bazel/_bazel_root/21c31a8bad1d906648b99f4a66de0f0f/external/rules_python/python/pip.bzl:47:10: pip_install is deprecated. Please switch to pip_parse. pip_install will be removed in a future release.
INFO: Analyzed target //:envoy (883 packages loaded, 35928 targets configured).
INFO: Found 1 target...
...
INFO: Elapsed time: 122.337s, Critical Path: 59.03s
INFO: 12805 processes: 7826 remote cache hit, 4976 internal, 1 local, 2 remote.
INFO: Build completed successfully, 12805 total actions

real    2m2.435s
user    0m0.001s
sys     0m0.418s

# echo $?
0

Also, I'm confused with Starting local Bazel server and connecting to it... message in output above - this is specific to envoy, as "Hello world!" remote execution doesn't emit this message...

Is it bazelisk, bazel, envoy or buildfarm? In my opinion it either bazelisk or bazel - the tool should not crash. As to buildfarm - in observed case it seems to be unrelated, as bazelisk crashes before anything is sent to remote computer. But also it could be specific to envoy, as it has really complicated build process.

Expected behavior: Remote execution build can be run and successfully completed without crashes and/or necessity to specify addition flags, like --profile