go-grpc-middleware
go-grpc-middleware copied to clipboard
recovery doesn't work with prometheus
I'm trying to get prometheus to report recovered panics, but it seems it doesn't work, regardless of middleware order.
Simple example:
opts := []grpc.ServerOption{
grpc_middleware.WithStreamServerChain(
grpc_recovery.StreamServerInterceptor(),
grpc_prometheus.StreamServerInterceptor,
),
grpc_middleware.WithUnaryServerChain(
grpc_recovery.UnaryServerInterceptor(),
grpc_prometheus.UnaryServerInterceptor,
),
}
grpcServer := grpc.NewServer(opts...)
myService.RegisterMyServiceServer(grpcServer, &myServer{})
grpc_prometheus.Register(grpcServer)
// ... handle /metrics for prometheus, serve grpc server
When myService.MyMethod
panics, it correctly returns a grpc internal error, but prometheus reads grpc_server_handled_total{grpc_code="Internal",grpc_method="MyMethod",grpc_service="myService.MyService",grpc_type="unary"} 0
I know prometheus is working, because grpc_server_msg_received_total
shows the correct count - it's just missing from the grpc_server_handled_total
metrics.
Is there a way for the recovery to return an error which is then used by other middleware? I've tried swapping the order of recovery and prometheus, but got the same result.