Investigate integration test failures when all tests run together

[michaelkaye]

Mar 08, 2022 2:26:04 PM com.google.testing.platform.RunnerImpl$Companion summarizeAndLogTestResult
SEVERE: Execute org.matrix.android.sdk.internal.crypto.keysbackup.KeysBackupTest.testBackupWhenAnotherBackupWasCreated: FAILED
java.lang.AssertionError: java.lang.AssertionError: expected:<WrongBackUpVersion> but was:<ReadyToBackUp>
at org.junit.Assert.fail(Assert.java:88)
at org.junit.Assert.failNotEquals(Assert.java:834)
at org.junit.Assert.assertEquals(Assert.java:118)
at org.junit.Assert.assertEquals(Assert.java:144)
at org.matrix.android.sdk.internal.crypto.keysbackup.KeysBackupTest.testBackupWhenAnotherBackupWasCreated(KeysBackupTest.kt:981)
java.lang.AssertionError: expected:<WrongBackUpVersion> but was:<ReadyToBackUp>
at org.junit.Assert.fail(Assert.java:88)
at org.junit.Assert.failNotEquals(Assert.java:834)
at org.junit.Assert.assertEquals(Assert.java:118)
at org.junit.Assert.assertEquals(Assert.java:144)
at org.matrix.android.sdk.internal.crypto.keysbackup.KeysBackupTest.testBackupWhenAnotherBackupWasCreated(KeysBackupTest.kt:981)

Mar 08, 2022 2:26:04 PM com.google.testing.platform.RunnerImpl$Companion summarizeAndLogTestResult
SEVERE: Execute org.matrix.android.sdk.session.space.SpaceHierarchyTest.testParentRelation: FAILED
java.lang.AssertionError
at org.junit.Assert.fail(Assert.java: java.lang.AssertionError
at org.junit.Assert.fail(Assert.java:86)
at org.junit.Assert.assertTrue(Assert.java:41)
at org.junit.Assert.assertTrue(Assert.java:52)
at org.matrix.android.sdk.common.CommonTestHelper.await(CommonTestHelper.kt:378)
at org.matrix.android.sdk.common.CommonTestHelper.waitWithLatch(CommonTestHelper.kt:396)
at org.matrix.android.sdk.common.CommonTestHelper.waitWithLatch$default(CommonTestHelper.kt:391)
at org.matrix.android.sdk.session.space.SpaceHierarchyTest.createPrivateSpace(SpaceHierarchyTest.kt:410)
at org.matrix.android.sdk.session.space.SpaceHierarchyTest.testParentRelation(SpaceHierarchyTest.kt:491)
java.lang.AssertionError
at org.junit.Assert.fail(Assert.java:86)
at org.junit.Assert.assertTrue(Assert.java:41)
at org.junit.Assert.assertTrue(Assert.java:52)
at org.matrix.android.sdk.common.CommonTestHelper.await(CommonTestHelper.kt:378)
at org.matrix.android.sdk.common.CommonTestHelper.waitWithLatch(CommonTestHelper.kt:396)
at org.matrix.android.sdk.common.CommonTestHelper.waitWithLatch$default(CommonTestHelper.kt:391)
at org.matrix.android.sdk.session.space.SpaceHierarchyTest.createPrivateSpace(SpaceHierarchyTest.kt:410)
at org.matrix.android.sdk.session.space.SpaceHierarchyTest.testParentRelation(SpaceHierarchyTest.kt:491)

Mar 08, 2022 2:26:04 PM com.google.testing.platform.RunnerImpl$Companion summarizeAndLogTestResult
SEVERE: Execute org.matrix.android.sdk.session.space.SpaceHierarchyTest.testRootSpaces: FAILED
java.lang.AssertionError
at org.junit.Assert.fail(Assert.java: java.lang.AssertionError
at org.junit.Assert.fail(Assert.java:86)
at org.junit.Assert.assertTrue(Assert.java:41)
at org.junit.Assert.assertTrue(Assert.java:52)
at org.matrix.android.sdk.common.CommonTestHelper.await(CommonTestHelper.kt:378)
at org.matrix.android.sdk.common.CommonTestHelper.waitWithLatch(CommonTestHelper.kt:396)
at org.matrix.android.sdk.common.CommonTestHelper.waitWithLatch$default(CommonTestHelper.kt:391)
at org.matrix.android.sdk.session.space.SpaceHierarchyTest.createPublicSpace(SpaceHierarchyTest.kt:382)
at org.matrix.android.sdk.session.space.SpaceHierarchyTest.testRootSpaces(SpaceHierarchyTest.kt:445)
java.lang.AssertionError
at org.junit.Assert.fail(Assert.java:86)
at org.junit.Assert.assertTrue(Assert.java:41)
at org.junit.Assert.assertTrue(Assert.java:52)
at org.matrix.android.sdk.common.CommonTestHelper.await(CommonTestHelper.kt:378)
at org.matrix.android.sdk.common.CommonTestHelper.waitWithLatch(CommonTestHelper.kt:396)
at org.matrix.android.sdk.common.CommonTestHelper.waitWithLatch$default(CommonTestHelper.kt:391)
at org.matrix.android.sdk.session.space.SpaceHierarchyTest.createPublicSpace(SpaceHierarchyTest.kt:382)
at org.matrix.android.sdk.session.space.SpaceHierarchyTest.testRootSpaces(SpaceHierarchyTest.kt:445)

The above three failures occurred when running :matrix-android-sdk:connectedDebugAndroidTest with no package/class limits, in both CI and locally on my desktop.

The tests did not individually fail when running in five separate package-scoped limited runs, so there is some interaction between them (or perhaps between a previous test and them) that is causing this issue.

[michaelkaye]

These look like timeout issues; increasing the timeout from 30s to 60s seems to fix all these tests.

I wonder if the reason splitting these tests into four was nothing more than allowing the synapse process some time to recover between runs.

It looks like others are considering a similar change: https://github.com/vector-im/element-android/pull/5380/files/3c931d6f6dc809ed6a812e8cf7d8e14c671b85c4#diff-4151eb8a0d2de530c600c7996c99e67e47bbcc32e59a56a73485fbe63e171a15

[michaelkaye]

I'm going to investigate a few things:

• Merge the change to increase the timeout to 60s will make the tests runnable in a large batch.
• Adding some sort of timer metric to see how much time is left on each of these latch awaits; identifying if there's specific calls that are being slow.
• Work out how to scrape metrics from synapse (locally at least) as we go, so we can identify if we are being blocked in the synapse in some way.