log message overload in batchRecvLoop connection failure

[mjonss]

See https://github.com/pingcap/tidb/issues/29190

1. Minimal reproduce step (Required) I set tikv-server security.cert-allowed-cn to a value that does not match tidb-server's certificate CN.

2. What did you expect to see? (Required) This should result in some managed failure where tidb-server exits or attempts reconnections with some kind of sensible backoff.

3. What did you see instead (Required) tidb-server floods its log output with messages, writing up to dozens of messages every ms.

[2021/10/27 21:41:27.871 +00:00] [INFO] [client_batch.go:565] ["batchRecvLoop re-create streaming success"] [target=127.0.0.1:20160] [forwardedHost=]
[2021/10/27 21:41:27.871 +00:00] [INFO] [client_batch.go:601] ["batchRecvLoop fails when receiving, needs to reconnect"] [target=127.0.0.1:20160] [forwardedHost=] [error="rpc error: code = Unauthenticated desc = Common name check fail, reason: x509_common_name from peer is tidb, cert_allowed_cn: {\"nonsense\"}"]
[2021/10/27 21:41:27.871 +00:00] [INFO] [client_batch.go:601] ["batchRecvLoop fails when receiving, needs to reconnect"] [target=127.0.0.1:20160] [forwardedHost=] [error="rpc error: code = Unauthenticated desc = Common name check fail, reason: x509_common_name from peer is tidb, cert_allowed_cn: {\"nonsense\"}"]
[2021/10/27 21:41:27.871 +00:00] [INFO] [client_batch.go:565] ["batchRecvLoop re-create streaming success"] [target=127.0.0.1:20160] [forwardedHost=]
[2021/10/27 21:41:27.871 +00:00] [INFO] [client_batch.go:601] ["batchRecvLoop fails when receiving, needs to reconnect"] [target=127.0.0.1:20160] [forwardedHost=] [error="rpc error: code = Unauthenticated desc = Common name check fail, reason: x509_common_name from peer is tidb, cert_allowed_cn: {\"nonsense\"}"]

4. What is your TiDB version? (Required)

tidb_version(): Release Version: v5.2.1
Edition: Community
Git Commit Hash: cd8fb24c5f7ebd9d479ed228bb41848bd5e97445
Git Branch: heads/refs/tags/v5.2.1
UTC Build Time: 2021-09-08 02:32:56
GoVersion: go1.16.4
Race Enabled: false
TiKV Min Version: v3.0.0-60965b006877ca7234adaced7890d7b029ed1306
Check Table Before Drop: false

Which should use:

        github.com/tikv/client-go/v2 v2.0.0-alpha.0.20210820060448-daddf73a0706

[mjonss]

Could possibly be improved by adding log sampling (rate limiting) in the logging configuration, like https://github.com/pingcap/tidb/pull/29362 .