Flaky-test: DelayedDeliveryTest.testDelayedDeliveryWithMultipleConcurrentReadEntries

[michaeljmarshall]

Search before asking

• [X] I searched in the issues and found nothing similar.

Example failure

https://github.com/apache/pulsar/runs/7753038071?check_suite_focus=true

Exception stacktrace

  Error:  testDelayedDeliveryWithMultipleConcurrentReadEntries(org.apache.pulsar.broker.service.persistent.DelayedDeliveryTest)  Time elapsed: 6.726 s  <<< FAILURE!
  java.lang.AssertionError: expected [1000] but found [998]
  	at org.testng.Assert.fail(Assert.java:99)
  	at org.testng.Assert.failNotEquals(Assert.java:1037)
  	at org.testng.Assert.assertEqualsImpl(Assert.java:140)
  	at org.testng.Assert.assertEquals(Assert.java:122)
  	at org.testng.Assert.assertEquals(Assert.java:907)
  	at org.testng.Assert.assertEquals(Assert.java:917)
  	at org.apache.pulsar.broker.service.persistent.DelayedDeliveryTest.testDelayedDeliveryWithMultipleConcurrentReadEntries(DelayedDeliveryTest.java:283)
  	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
  	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
  	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
  	at java.base/java.lang.reflect.Method.invoke(Method.java:568)
  	at org.testng.internal.MethodInvocationHelper.invokeMethod(MethodInvocationHelper.java:132)
  	at org.testng.internal.InvokeMethodRunnable.runOne(InvokeMethodRunnable.java:45)
  	at org.testng.internal.InvokeMethodRunnable.call(InvokeMethodRunnable.java:73)
  	at org.testng.internal.InvokeMethodRunnable.call(InvokeMethodRunnable.java:11)
  	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
  	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
  	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
  	at java.base/java.lang.Thread.run(Thread.java:833)

Are you willing to submit a PR?

• [X] I'm willing to submit a PR!

[michaeljmarshall]

It looks like the test is failing sometimes because the consumer is getting duplicate messages. This flakiness may be an actual bug. I'm able to reproduce this flaky test failure when running on my local machine.

[michaeljmarshall]

Both #16812 and #16609 look related since they modified the readMoreEntries method used heavily in this test.

[michaeljmarshall]

#16968 also looks related.