go-grpc-middleware icon indicating copy to clipboard operation
go-grpc-middleware copied to clipboard

recovery doesn't work with prometheus

Open frostbyte73 opened this issue 5 years ago • 0 comments

I'm trying to get prometheus to report recovered panics, but it seems it doesn't work, regardless of middleware order.

Simple example:

opts := []grpc.ServerOption{
	grpc_middleware.WithStreamServerChain(
		grpc_recovery.StreamServerInterceptor(),
		grpc_prometheus.StreamServerInterceptor,
	),
	grpc_middleware.WithUnaryServerChain(
		grpc_recovery.UnaryServerInterceptor(),
		grpc_prometheus.UnaryServerInterceptor,
	),
}

grpcServer := grpc.NewServer(opts...)
myService.RegisterMyServiceServer(grpcServer, &myServer{})
grpc_prometheus.Register(grpcServer)
// ... handle /metrics for prometheus, serve grpc server

When myService.MyMethod panics, it correctly returns a grpc internal error, but prometheus reads grpc_server_handled_total{grpc_code="Internal",grpc_method="MyMethod",grpc_service="myService.MyService",grpc_type="unary"} 0

I know prometheus is working, because grpc_server_msg_received_total shows the correct count - it's just missing from the grpc_server_handled_total metrics.

Is there a way for the recovery to return an error which is then used by other middleware? I've tried swapping the order of recovery and prometheus, but got the same result.

frostbyte73 avatar May 29 '19 00:05 frostbyte73