Bazel builds manifest InvalidArgument server-side

[rejuvenile]

This occurs occasionally:

2025-03-03T17:26:44.974998Z ERROR nativelink_service::bytestream_server: error: status: InvalidArgument, message: "Expected WriteRequest struct in stream : In ByteStreamServer::write", details: [], metadata: MetadataMap { headers: {} }
    at nativelink-service/src/bytestream_server.rs:641
    in nativelink_service::bytestream_server::write with request: Streaming
    in nativelink_util::task::http_executor
    in nativelink::services::http_connection with remote_addr: 192.168.191.244:56972, socket_addr: 0.0.0.0:50051

[aaronmondal]

This sounds like a toolchain issue. NativeLink forwards "raw commands" to workers. This means that if such a command references a path like /usr/bin/someexecutable it'll invoke whatever is present on the worker at that location. Without fairly detailed info about your setup it's hard to figure out what exactly the issue is. If this is the same setup as #1607 my guess is that you're sending commands from a MacOS toolchain to:

• A linux executor. In this case the unfortunate news is that regular rules_rust IMO doesn't correctly support crosscompilation. You'd have to set up your own crosscompilation toolchains or use something like our lre-rs toolchain https://github.com/TraceMachina/nativelink/tree/main/local-remote-execution
• Another MacOS system. In this case you'd have to make sure that you install or override whatever toolchain is on the other mac. This kind of issue often points to hermeticity issues (rules_rust is not hermetic by default). An ad-hoc workaround might be to manually try to sync up your worker to the host.

Note that if you run nativelink via a container on macos it'll effectively run in a Linux VM. So if you use a Mac worker make sure to run nativelink as raw binary directly on the mac without any containerization. (I.e. via nix run github:TraceMachina/nativelink).

[rejuvenile]

I have not seen a repeat since toolchain issues resolved.

[rejuvenile]

Alas, I still see this, even with a correctly configured toolchain:

2025-03-12T20:12:54.727079Z ERROR nativelink_service::bytestream_server: error: status: InvalidArgument, message: "Expected WriteRequest struct in stream : In ByteStreamServer::write", details: [], metadata: MetadataMap { headers: {} }
    at nativelink-service/src/bytestream_server.rs:641
    in nativelink_service::bytestream_server::write with request: Streaming
    in nativelink_util::task::http_executor
    in nativelink::services::http_connection with remote_addr: 192.168.191.244:63694, socket_addr: 0.0.0.0:50051

  2025-03-12T20:12:54.803699Z ERROR nativelink_service::bytestream_server: error: status: Internal, message: "status: Internal, message: \"Unexpected EOF decoding stream.\", details: [], metadata: MetadataMap { headers: {} } : Stream error at byte 753664 : In ByteStreamServer::write", details: [], metadata: MetadataMap { headers: {} }
    at nativelink-service/src/bytestream_server.rs:641
    in nativelink_service::bytestream_server::write with request: Streaming
    in nativelink_util::task::http_executor
    in nativelink::services::http_connection with remote_addr: 192.168.191.244:63695, socket_addr: 0.0.0.0:50051

[rejuvenile]

Also:

  2025-03-12T20:17:42.148263Z ERROR nativelink_service::bytestream_server: error: status: ResourceExhausted, message: "status: ResourceExhausted, message: \"h2 protocol error: error reading a body from connection\", details: [], metadata: MetadataMap { headers: {} } : Expected WriteRequest struct in stream : Could not unwrap first stream message", details: [], metadata: MetadataMap { headers: {} }
    at nativelink-service/src/bytestream_server.rs:641
    in nativelink_service::bytestream_server::write with request: Streaming
    in nativelink_util::task::http_executor
    in nativelink::services::http_connection with remote_addr: 192.168.191.244:64617, socket_addr: 0.0.0.0:50051