hyperium
Received 'Status::internal("Missing request message.")' in case of cancellation by client
Bug Report
Version
Verified it in tonic 0.12 and 0.14.
Platform
- 24.6.0 Darwin Kernel
- Verified on linux as well.
Crates
tonic
Description
If the cancellation is sent from the client before the full request message reached the server, it leads to Status::internal("Missing request message.").
I reproduced this issue locally by java gRPC client (with hedging) and tonic server with request size of around 1 MB.
Error is thrown here.
Expected Behaviour:
- Case1: If the cancellation is sent from the client before the full request message reached the server, then request future must be dropped and no response/error should be returned to the tower layers.
- Case2: If the cancellation is sent from the client after the full request message reached the server, then request future must be dropped and no response/error should be returned to the tower layers. This is working as expected.
I'm not sure if this is the intended behaviour that needs to be handled by the server. In my opinion, case 1 should drop the future instead of return Status::internal("Missing request message.").
@madhurishgupta where are you seeing the "Missing request message" error manifest? The h2 layer should see the cancellation and not send anything back to the client when this occurs. And even if it didn't, the client shouldn't be using any returned status, either, since the client initiated cancelation. To me this scenario seems like something that should only have any effect internally and not be visible to the application. It might be helpful if you could provide a small reproduction example showing the error. Thank you.
@dfawley thanks for the prompt reply.
Yes, the error Status::internal("Missing request message.") is not sent back to the client. But this error is propagated to all the tower layers on the server side (metrics layer, trace layer etc).
For reproduction, I used java gRPC client with hedging delay of 1 second and tonic server in which handler implementation has sleep of 900 ms. Then I sent the request of size 1 MB to the server. Below is the flow:
- Request 1 starts from the client at time=0 ms.
- Request 1 goes to sleep on the server side for 900 ms.
- Due to hedging, the request 2 starts from the client after time = 1 second.
- After few milliseconds, response of request 1 reaches the client and it cancels the request 2.
- Server receives the cancellation of request 2 but full request 2 didn't reach the server, instead of dropping the request, it throws
Status::internal("Missing request message.")here during deserialization of the message body.
Below are the logs of trace layer on server side:
2025-09-05T17:28:10.799777Z DEBUG request{method=POST uri=http://localhost:50051/helloworld.Greeter/SayHello version=HTTP/2.0}: tower_http::trace::on_request: started processing request
2025-09-05T17:28:11.716159Z DEBUG request{method=POST uri=http://localhost:50051/helloworld.Greeter/SayHello version=HTTP/2.0}: tower_http::trace::on_response: finished processing request latency=916 ms response_headers={"content-type": "application/grpc"}
2025-09-05T17:28:11.720559Z DEBUG request{method=POST uri=http://localhost:50051/helloworld.Greeter/SayHello version=HTTP/2.0}: tower_http::trace::on_eos: end of stream stream_duration=4 ms status=0
2025-09-05T17:28:11.737302Z DEBUG request{method=POST uri=http://localhost:50051/helloworld.Greeter/SayHello version=HTTP/2.0}: tower_http::trace::on_request: started processing request
2025-09-05T17:28:11.737920Z DEBUG request{method=POST uri=http://localhost:50051/helloworld.Greeter/SayHello version=HTTP/2.0}: tower_http::trace::on_response: finished processing request latency=0 ms status=13 response_headers={"content-type": "application/grpc", "grpc-status": "13", "grpc-message": "Missing%20request%20message."}
2025-09-05T17:28:11.737998Z ERROR request{method=POST uri=http://localhost:50051/helloworld.Greeter/SayHello version=HTTP/2.0}: tower_http::trace::on_failure: response failed classification=Code: 13 latency=0 ms
My hunch is that tonic is trying to deserialize the RST_STREAM (cancellation from client) which is leading to this behavior.
