client-go icon indicating copy to clipboard operation
client-go copied to clipboard
verified publisher icon tikv

The metric `tidb_tikvclient_batch_recv_latency` is not reasonable

Open you06 opened this issue 3 years ago • 1 comments

Found by @zyguan

Start a cluster and wait a while, then run some workload, we got "Batch Receive Average Duration" panel like this.

image

The way we observe the receiving time is not reasonable. In an idle cluster, this function is waiting for responses a long time while there are no in-fly requests, so the receiving time is high. In a busy cluster, the responses are pending in the client's buffer and we just loop reading them from memory, so the receiving time is extremely low(avg 547us).

https://github.com/tikv/client-go/blob/79a0fc59e4d5012095282344f5fdbca1492de771/internal/client/client_batch.go#L427-L449

you06 avatar Jul 05 '22 09:07 you06

maybe change it to another title? such as 'Batch Wait for response duration'?

disksing avatar Jul 25 '22 04:07 disksing