async_grpc icon indicating copy to clipboard operation
async_grpc copied to clipboard

shutdown server while client is still connected

Open flixr opened this issue 5 years ago • 3 comments

Is there a way to cleanly shutdown the server while there are still clients connected (i.e. finish the connections)? I could not find any method/callback for this.. Am I missing something or is this not implemented?

When I try to shutdown the server and a RpcHandler is still running (because the client didn't disconnect yet), it will just hang forever. If I then shutdown the client I get a failure with RPCs still in flight:

I1130 16:54:18.130954     7 server.cc:215] Shutting down server.

-> hangs here until I shutdown the client and then fails:

I1130 16:56:32.376948     7 completion_queue_thread.cc:39] Shutting down completion queue 0x557085fd3a40
I1130 16:56:32.377357     7 completion_queue_thread.cc:39] Shutting down completion queue 0x557085fc8f20
I1130 16:56:32.377645     7 event_queue_thread.cc:38] Shutting down event queue 0x557085fcc400
I1130 16:56:32.377735     7 event_queue_thread.cc:38] Shutting down event queue 0x557085fcbb00
I1130 16:56:32.377812     7 server.cc:237] Shutdown complete.
F1130 16:56:32.383496     7 rpc.cc:355] RPCs still in flight!

It would be nice to have a OnShutdown callback in the RpcHandler (where the handler could then finish the write and hence shutdown the connection to the client). Or should this be done e.g. by registering a custom callback via ExecutionContext?

flixr avatar Nov 30 '20 17:11 flixr

I came across this too recently, I ended up adding a Rpc::TryCancel method and calling it from Service::StopServing.

Something along the lines of:

void Rpc::TryCancel() {
  if (IsRpcEventPending(Rpc::Event::READ) ||
      IsRpcEventPending(Rpc::Event::WRITE) ||
      IsRpcEventPending(Rpc::Event::FINISH)) {
    LOG(TRACE) << "Rpc::TryCancel for " << (void*)this;
    server_context_.TryCancel();
  }
}

which is called from a new method on ActiveRpcs:

void ActiveRpcs::TryCancel() {
  common::MutexLocker locker(&lock_);
  for (auto it : rpcs_) {
    it.second->TryCancel();
  }
}

which in turn is called from Service!

void Service::StopServing() {
  shutting_down_ = true;
  active_rpcs_.TryCancel();
}

TobyEalden avatar Dec 01 '20 13:12 TobyEalden

Hm... interesting.. I'm not very familiar with gRPC (yet), but it says

it is generally preferred to explicitly finish an RPC by returning Status::CANCELLED rather than using TryCancel

so I added a OnServerShutdown callback where Finish(Status::CANCELLED) can be called. See #58 Feedback would be very much appreciated ;-)

flixr avatar Dec 01 '20 18:12 flixr

Neither the TryCancel nor the OnServerShutdown (at least as done in #58) works.

I have a working solution through custom callback via execution context, but it would be nice to have this supported by default.

flixr avatar Dec 02 '20 09:12 flixr