Allow clients to send data before headers are received

[blinsay]

trafficstars

Is your feature request related to a problem? Please describe.

tonic wraps hyper to provide request/response and bidirectional streaming for GRPC over http/2. When using a bidirectional streaming client, tonic waits until it receives response headers to return control to the client. That means that straight line code like this can hang:

    let mut client = EchoClient::connect("http://127.0.0.1:50051").await.unwrap();

    let (tx, rx) = tokio::sync::mpsc::channel(10);
    let response = client
        .bidirectional_streaming_echo(tokio_stream::wrappers::ReceiverStream::new(rx))
        .await
        .unwrap();

    for i in 0..10 {
        tx.send(EchoRequest {
            message: format!("msg {:02}", i),
        })
        .await
        .unwrap();
    }

    let mut resp_stream = response.into_inner();
    while let Some(received) = resp_stream.next().await {
        let received = received.unwrap();
        println!("\treceived message: `{}`", received.message);
    }

Internally, the bidirectional_streaming_* call generates a hyper::client::conn::http2::SendRequest and awaits it (see tonic's SendRequest. Waiting on a SendRequest::send_request appears to send headers from the client and block until it has read headers from the server.

In the example above, because the body stream is still pending data, the client sends http/2 headers to the server with no body and waiting for response headers from the server. If the server waits on replying with headers until it receives body data - which GRPC servers in Java, Go, and C++ appear to do - the SendRequest::send_request future will never complete, control will never return to the caller so no data will be sent, and everything will hang.

Describe alternatives you've considered

h2 exposes an API in SendRequest that returns control as soon as headers have been sent - the second example in the documentation is exactly the behavior that tonic wants to have. In talking with @LucioFranco in Discord it seems like tonic has a few alternative ways to go about getting that kind of behavior:

• Expose an h2-style API in hyper, and switch tonic to that API.
• Drop down to using h2 directly in tonic clients.
• Write code in tonic to get back to this behavior without relying on the transport.

Based on our discussions so far, there's some preference for the first option and we thought we'd start a discussion here.