yawp
yawp copied to clipboard
Flaky test fix
What changes were proposed in this pull request?
I observed a previous suggested fix injected some delay to fix the flaky test, but I believe that fix might be unstable in a CI environment or when run on different machines, given the dependency on some constant wait time. I suggest a new way to fix the test by adding some synchronization for the test execution only. I at first identify the source code location whose slow execution leads to the flaky test failure, where if yawp-appengine/src/main/java/io/yawp/driver/appengine/pipes/flow/drops/FlowDropsTask.java#64 slows the execution in joining into the workList, then the test fails (yawp-testing/yawp-testing-appengine/src/main/java/io/yawp/testing/appengine/AsyncHelper.java#20). Hence, I introduce one variable in this class [FlowDropsTask.java] that is only there to provide some synchronization. Basically, until this statement executed, I force the thread that the test runs on to wait before it executes the assertions at Line 16. The waiting location is at yawp-testing/yawp-testing-appengine/src/main/java/io/yawp/testing/appengine/AsyncHelper.java#16
Why are the changes needed?
This test is flakily fails. I run this test many times and it makes assertion fails.
I was running this test many times and it fails. The failure message is as follows.
Failure: Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 34.768 sec <<< FAILURE! testFlowDropsToSameSink(io.yawp.testing.appengine.pipes.flow.FlowDropsTest) Time elapsed: 34.74 sec <<< ERROR! java.lang.RuntimeException: await timout at io.yawp.testing.appengine.AsyncHelper.awaitAsync(AsyncHelper.java:20) at io.yawp.testing.appengine.AppengineTestHelper.awaitAsync(AppengineTestHelper.java:54) at io.yawp.testing.EndpointTestCaseBase.awaitAsync(EndpointTestCaseBase.java:334) at io.yawp.testing.appengine.pipes.flow.FlowDropsTest.testFlowDropsToSameSink(FlowDropsTest.java:35) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:271) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:70) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:50) at org.junit.runners.ParentRunner$3.run(ParentRunner.java:238) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:63) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:236) at org.junit.runners.ParentRunner.access$000(ParentRunner.java:53) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:229) at org.junit.runners.ParentRunner.run(ParentRunner.java:309) at org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:252) at org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:141) at org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:112) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.maven.surefire.util.ReflectionUtils.invokeMethodWithArray(ReflectionUtils.java:189) at org.apache.maven.surefire.booter.ProviderFactory$ProviderProxy.invoke(ProviderFactory.java:165) at org.apache.maven.surefire.booter.ProviderFactory.invokeProvider(ProviderFactory.java:85) at org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:115) at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:75)
Apr 01, 2023 1:09:58 AM com.google.appengine.api.datastore.dev.LocalDatastoreService cleanupActiveServices INFO: scheduler shutting down.
Results :
Tests in error: testFlowDropsToSameSink(io.yawp.testing.appengine.pipes.flow.FlowDropsTest): await timout
Tests run: 1, Failures: 0, Errors: 1, Skipped: 0
Result:
With my changes, I run the test 1000 times and every time it passes the test.