Flaky-test: ManagedCursorMetricsTest. testManagedCursorMetrics

[RobertIndie]

ManagedCursorMetricsTest. testManagedCursorMetrics is flaky. It fails sporadically.

https://github.com/apache/pulsar/pull/16878 didn't fix the issue. The problem still exists in the latest code.

example failure

  Error:  testCursorReadWriteMetrics(org.apache.pulsar.broker.stats.ManagedCursorMetricsTest)  Time elapsed: 0.449 s  <<< FAILURE!
  java.lang.AssertionError: did not expect [0] but found [0]
  	at org.testng.Assert.fail(Assert.java:99)
  	at org.testng.Assert.failEquals(Assert.java:1041)
  	at org.testng.Assert.assertNotEqualsImpl(Assert.java:147)
  	at org.testng.Assert.assertNotEquals(Assert.java:1531)
  	at org.testng.Assert.assertNotEquals(Assert.java:1535)
  	at org.apache.pulsar.broker.stats.ManagedCursorMetricsTest.testCursorReadWriteMetrics(ManagedCursorMetricsTest.java:246)

[RobertIndie]

From the test log we can see that there are many zookeeper connection exceptions during the test:

2022-08-04T07:27:04,903 - INFO  - [main-SendThread(127.0.0.1:45483):ClientCnxn$SendThread@1171] - Opening socket connection to server localhost/127.0.0.1:45483.
2022-08-04T07:27:04,903 - INFO  - [main-SendThread(127.0.0.1:45483):ClientCnxn$SendThread@1173] - SASL config status: Will not attempt to authenticate using SASL (unknown error)
2022-08-04T07:27:04,904 - WARN  - [main-SendThread(127.0.0.1:45483):ClientCnxn$SendThread@1292] - Session 0x100000c35b70002 for server localhost/127.0.0.1:45483, Closing socket connection. Attempting reconnect except it is a SessionExpiredException.
java.net.ConnectException: Connection refused
	at sun.nio.ch.Net.pollConnect(Native Method) ~[?:?]
	at sun.nio.ch.Net.pollConnectNow(Net.java:672) ~[?:?]
	at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:946) ~[?:?]
	at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:344) ~[zookeeper-3.8.0.jar:3.8.0]
	at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1282) ~[zookeeper-3.8.0.jar:3.8.0]

[coderzc]

+1 https://github.com/apache/pulsar/runs/7718272297?check_suite_focus=true

[poorbarcode]

Hi @RobertIndie , @coderzc

https://github.com/apache/pulsar/pull/16878 didn't fix the issue. The problem still exists in the latest code. https://github.com/apache/pulsar/runs/7718272297?check_suite_focus=true

#16878 fixed testManagedCursorMetrics, the log you provided is another method testCursorReadWriteMetrics, the title of this issue should change to "Flaky-test: ManagedCursorMetricsTest.testCursorReadWriteMetrics"