[Bug] The metric `grpc_open` sometime incorrect

[xianjingfeng]

We found the value of grpc_open sometime very big(>1000) even no application run in our cluster

[jerqi]

Could you provide more detailed information? Could you add some logs to help us solve this problem?

[xianjingfeng]

No logs, we just found this phenomenon. Maybe org.apache.uniffle.common.rpc.MonitoringServerCall#close not called sometimes. I try to call decCounter in MonitoringServerCallListener#onComplete/onCancel/onComplete and it work. But i don't know the real reasion

[jerqi]

No logs, we just found this phenomenon. Maybe org.apache.uniffle.common.rpc.MonitoringServerCall#close not called sometimes. I try to call decCounter in MonitoringServerCallListener#onComplete/onCancel/onComplete and it work. But i don't know the real reasion

I guess that method close could not be called sometimes if there is an exception.

[jerqi]

cc @colinmjj , Do you remember our flaky metric test? I guess that it's caused by this issue.