userver icon indicating copy to clipboard operation
userver copied to clipboard

gRPC server hangs during shutdown with active bidirectional stream RPC

Open dmitryikh opened this issue 2 years ago • 1 comments
trafficstars

I can't stop grpc server implemented in userver because of the active client stream call.

Example of Grpc method handler:

void SomeService::SomeMethod(SomeMethodCall& call)
{

	SomeRequest request;
	while (call.Read(request))     <--- handling task is waiting in this method
	{
        SomeResponse;
        call.Write(response);
	}

	call.Finish();
}

Is there a way to cancel a stream RPC (gracefully or not) during the shutdown on the server side?

Digging deeper I found that the Read is non cancellable: https://github.com/userver-framework/userver/blob/develop/grpc/src/ugrpc/server/impl/async_method_invocation.cpp#L30-L40

That might be the root cause.

Any thought?

dmitryikh avatar Nov 17 '23 14:11 dmitryikh

At least one of the causes is that ugrpc::server::Server::Stop calls grpc::Server::Shutdown without a deadline argument, which means that existing server RPCs are not forcefully closed at any point. We need to either a reasonable grace period to grpc::Server::Shutdown. The period should be configurable in the static config.

Anton3 avatar Dec 04 '23 09:12 Anton3