Parallel async calls are being sent sequential by default

[senthilkumarv]

Test Repo Describe the bug I have gRPC server and client written using grpc-rs. I am trying to make parallel calls to the server from the client. I expect the calls to be fired in parallel however the client seems to be sequencing them. The same implementation in go (https://github.com/senthilkumarv/grpc-rs-async/blob/master/main.go) gives me the desired output.

Here is the client code:

let async_replies: Vec<_> = (1..5).collect::<Vec<i32>>().iter().map(|index| {
    let env = Arc::new(Environment::new(5));
    let channel = ChannelBuilder::new(env).connect("127.0.0.1:50051");
    let client = GreeterClient::new(channel);
    let mut request = HelloRequest::new();
    request.name = format!("Request {}", index);
    let future = client.say_hello_async(&request)
      .into_future()
      .and_then(|client| client)
      .map(move |res| {
        let _ = client;
        println!("Reply from {}", res.message);
        res.message
      });
    Box::new(future)
  }).collect();
  let result = futures::future::join_all(async_replies)
    .wait()
    .unwrap();

Expected result (from server log)

Starting to serve Request 2
Starting to serve Request 1
Starting to serve Request 3
Starting to serve Request 4
Done servicing Request 3. Took 704ms
Done servicing Request 1. Took 705ms
Done servicing Request 4. Took 704ms
Done servicing Request 2. Took 705ms

Actual result

Starting to serve Request 3
Done servicing Request 3. Took 703ms
Starting to serve Request 2
Done servicing Request 2. Took 702ms
Starting to serve Request 4
Done servicing Request 4. Took 704ms
Starting to serve Request 1
Done servicing Request 1. Took 704ms

Workaround I was able to achieve what I needed, by forcing new sub channels by setting a different user agents string every time I created a channel, here. Is there a proper way to do this? Also why is the behavior different between go vs rust/grpc-core.

[BusyJay]

This is related to #179.

In summary, grpc c core will reuse the same completion queue for all requests from the same connection until exceeding the slots. It's a known problem in gRPC c core (grpc/grpc#15535), however it seems they don't have the plan to change the behavior in near future.

On the other hand, you should avoid doing any time consuming job inside the service method. You can use futures or thread pool to do the work and let completion queue focus on network related tasks.

by forcing new sub channels by setting a different user agents string every time I created a channel,

That's because gRPC c core reuse connection under the hook. When using different user agents string, it will consider it's not appropriate to reuse the same connection, so create a new one, which can utilize other completion queue.

[senthilkumarv]

@BusyJay Thanks for the response. I did come across this when I was googling for this issue. I see the that the C-core provides CreateCustomChannel method where you can pass channel arguments and make it look like different channel. I feel like this is cleaner than modifying user agent string per channel creation.

Is there a plan to expose this in grpc-rs?

[BusyJay]

Actually they are already supported. But they are very hacky and just for test purpose so they are hidden from docs.

https://github.com/pingcap/grpc-rs/blob/7c53f68b02cf56a5dbe042bb7d43556ac61de81a/src/channel.rs#L398-L416