Messy requestFlow race when ClientCall.start() triggers error

[bubenheimer]

trafficstars

grpc-kotlin 1.3.0

In a bidi streaming or client streaming scenario there is a race between ClientCall.start() and requests flow initialization which causes messy request flow subscription & cancellation behavior when the server closes the RPC right after ClientCall.start(). This is not an unusual scenario: for example, the server might close the connection with an error due to invalid RPC header parameters, like an expired session ID. I see that in this case, requests flow collection is quickly cancelled, in my case often leading to partial requests flow initialization, cancelling the flow right in the middle of initial collection. The result is a mess in the state of my client app. I have reworked my client code to guard against this scenario, but it requires very careful consideration of early failure scenarios to robustly address partial initialization.

Basic sample bidi usage on client:

val responseFlow = myrpc(requestsFlow)

I don't think grpc-kotlin should behave in this unpredictable and error-prone manner. The simple mental model that I (and likely many other grpc-kotlin users) generally follow is: (for bidi streaming)

1. app code collects response flow
2. requestFlow collection started by grpc-kotlin
3. RPC connection is established and requests sent to server
4. response flow collection receives data

Instead what grpc-kotlin does is:

1. app code collects response flow
2. RPC connection starts being established
3. Right afterwards, and concurrently, requestFlow collection is started
4. Responses are passed to collector

grpc-kotlin's behavior is an unexpected implementation detail. I generally think of requests flow subscription as the logical trigger to starting the RPC, but the current implementation does something else. Documentation does not provide a clear contract. I suppose it is arguable whether upon the client collecting the response flow, grpc-kotlin should be allowed to first start setting up the channel before subscribing to the requests flow. Regardless of the answer to this question, the current behavior causes a potentially messy race.

I don't really see an automatic, reliable way to guarantee full requestFlow initialization from grpc-kotlin code prior to RPC connection setup. Perhaps using ReceiveChannels instead of flows could be part of a solution: client app code could fully control initialization of the stream of requests before initiating the RPC. Of course, this would be a very major API and behavior change.

Again, I think that this is important to address because of the current racy, unpredictable, and error-prone behavior.