eclipse.platform Random failing ParallelBuildChainTest.testIndividualProjectBuilds_WithManyProjects

ParallelBuildChainTest.testIndividualProjectBuilds_WithManyProjects_ProjectRelaxedRule randomly fails on Linux I-Builds, e.g., https://download.eclipse.org/eclipse/downloads/drops4/I20231105-1800/testresults/html/org.eclipse.core.tests.resources_ep430I-unit-cen64-gtk3-java21_linux.gtk.x86_64_21.html

It is random because execution succeeds in some builds: https://download.eclipse.org/eclipse/downloads/drops4/I20231103-0420/testResults.php

The test fails after around 20 seconds, which is the timeout value in ParallelBuildChainTest, so maybe the test is only performing too bad. The test case has been introduced with #661.

Exemplary log:

not all build jobs have started in time Expected: (a collection containing <P/longRunningBuildProject0> and a collection containing <P/longRunningBuildProject1> and a collection containing <P/longRunningBuildProject10> and a collection containing <P/longRunningBuildProject11> and a collection containing <P/longRunningBuildProject12> and a collection containing <P/longRunningBuildProject13> and a collection containing <P/longRunningBuildProject14> and a collection containing <P/longRunningBuildProject15> and a collection containing <P/longRunningBuildProject16> and a collection containing <P/longRunningBuildProject17> and a collection containing <P/longRunningBuildProject18> and a collection containing <P/longRunningBuildProject19> and a collection containing <P/longRunningBuildProject2> and a collection containing <P/longRunningBuildProject20> and a collection containing <P/longRunningBuildProject21> and a collection containing <P/longRunningBuildProject22> and a collection containing <P/longRunningBuildProject23> and a collection containing <P/longRunningBuildProject24> and a collection containing <P/longRunningBuildProject25> and a collection containing <P/longRunningBuildProject26> and a collection containing <P/longRunningBuildProject27> and a collection containing <P/longRunningBuildProject28> and a collection containing <P/longRunningBuildProject29> and a collection containing <P/longRunningBuildProject3> and a collection containing <P/longRunningBuildProject30> and a collection containing <P/longRunningBuildProject31> and a collection containing <P/longRunningBuildProject32> and a collection containing <P/longRunningBuildProject33> and a collection containing <P/longRunningBuildProject34> and a collection containing <P/longRunningBuildProject35> and a collection containing <P/longRunningBuildProject36> and a collection containing <P/longRunningBuildProject37> and a collection containing <P/longRunningBuildProject38> and a collection containing <P/longRunningBuildProject39> and a collection containing <P/longRunningBuildProject4> and a collection containing <P/longRunningBuildProject40> and a collection containing <P/longRunningBuildProject41> and a collection containing <P/longRunningBuildProject42> and a collection containing <P/longRunningBuildProject43> and a collection containing <P/longRunningBuildProject44> and a collection containing <P/longRunningBuildProject45> and a collection containing <P/longRunningBuildProject46> and a collection containing <P/longRunningBuildProject47> and a collection containing <P/longRunningBuildProject48> and a collection containing <P/longRunningBuildProject49> and a collection containing <P/longRunningBuildProject5> and a collection containing <P/longRunningBuildProject50> and a collection containing <P/longRunningBuildProject51> and a collection containing <P/longRunningBuildProject52> and a collection containing <P/longRunningBuildProject53> and a collection containing <P/longRunningBuildProject54> and a collection containing <P/longRunningBuildProject55> and a collection containing <P/longRunningBuildProject56> and a collection containing <P/longRunningBuildProject57> and a collection containing <P/longRunningBuildProject58> and a collection containing <P/longRunningBuildProject59> and a collection containing <P/longRunningBuildProject6> and a collection containing <P/longRunningBuildProject7> and a collection containing <P/longRunningBuildProject8> and a collection containing <P/longRunningBuildProject9>) but: a collection containing <P/longRunningBuildProject10> mismatches were: [was <P/longRunningBuildProject0>, was <P/longRunningBuildProject12>, was <P/longRunningBuildProject22>, was <P/longRunningBuildProject34>, was <P/longRunningBuildProject40>, was <P/longRunningBuildProject50>, was <P/longRunningBuildProject37>, was <P/longRunningBuildProject59>, was <P/longRunningBuildProject16>, was <P/longRunningBuildProject57>, was <P/longRunningBuildProject39>, was <P/longRunningBuildProject47>, was <P/longRunningBuildProject7>, was <P/longRunningBuildProject13>, was <P/longRunningBuildProject24>, was <P/longRunningBuildProject2>, was <P/longRunningBuildProject19>, was <P/longRunningBuildProject8>, was <P/longRunningBuildProject55>, was <P/longRunningBuildProject6>, was <P/longRunningBuildProject18>, was <P/longRunningBuildProject42>, was <P/longRunningBuildProject58>, was <P/longRunningBuildProject52>, was <P/longRunningBuildProject33>, was <P/longRunningBuildProject54>, was <P/longRunningBuildProject17>, was <P/longRunningBuildProject30>, was <P/longRunningBuildProject38>, was <P/longRunningBuildProject56>, was <P/longRunningBuildProject23>, was <P/longRunningBuildProject41>, was <P/longRunningBuildProject25>, was <P/longRunningBuildProject31>, was <P/longRunningBuildProject45>, was <P/longRunningBuildProject14>, was <P/longRunningBuildProject15>, was <P/longRunningBuildProject35>, was <P/longRunningBuildProject32>, was <P/longRunningBuildProject1>, was <P/longRunningBuildProject48>, was <P/longRunningBuildProject3>, was <P/longRunningBuildProject27>, was <P/longRunningBuildProject53>, was <P/longRunningBuildProject44>, was <P/longRunningBuildProject49>, was <P/longRunningBuildProject29>, was <P/longRunningBuildProject4>, was <P/longRunningBuildProject28>, was <P/longRunningBuildProject11>, was <P/longRunningBuildProject20>]

java.lang.AssertionError: not all build jobs have started in time
Expected: (a collection containing <P/longRunningBuildProject0> and a collection containing <P/longRunningBuildProject1> and a collection containing <P/longRunningBuildProject10> and a collection containing <P/longRunningBuildProject11> and a collection containing <P/longRunningBuildProject12> and a collection containing <P/longRunningBuildProject13> and a collection containing <P/longRunningBuildProject14> and a collection containing <P/longRunningBuildProject15> and a collection containing <P/longRunningBuildProject16> and a collection containing <P/longRunningBuildProject17> and a collection containing <P/longRunningBuildProject18> and a collection containing <P/longRunningBuildProject19> and a collection containing <P/longRunningBuildProject2> and a collection containing <P/longRunningBuildProject20> and a collection containing <P/longRunningBuildProject21> and a collection containing <P/longRunningBuildProject22> and a collection containing <P/longRunningBuildProject23> and a collection containing <P/longRunningBuildProject24> and a collection containing <P/longRunningBuildProject25> and a collection containing <P/longRunningBuildProject26> and a collection containing <P/longRunningBuildProject27> and a collection containing <P/longRunningBuildProject28> and a collection containing <P/longRunningBuildProject29> and a collection containing <P/longRunningBuildProject3> and a collection containing <P/longRunningBuildProject30> and a collection containing <P/longRunningBuildProject31> and a collection containing <P/longRunningBuildProject32> and a collection containing <P/longRunningBuildProject33> and a collection containing <P/longRunningBuildProject34> and a collection containing <P/longRunningBuildProject35> and a collection containing <P/longRunningBuildProject36> and a collection containing <P/longRunningBuildProject37> and a collection containing <P/longRunningBuildProject38> and a collection containing <P/longRunningBuildProject39> and a collection containing <P/longRunningBuildProject4> and a collection containing <P/longRunningBuildProject40> and a collection containing <P/longRunningBuildProject41> and a collection containing <P/longRunningBuildProject42> and a collection containing <P/longRunningBuildProject43> and a collection containing <P/longRunningBuildProject44> and a collection containing <P/longRunningBuildProject45> and a collection containing <P/longRunningBuildProject46> and a collection containing <P/longRunningBuildProject47> and a collection containing <P/longRunningBuildProject48> and a collection containing <P/longRunningBuildProject49> and a collection containing <P/longRunningBuildProject5> and a collection containing <P/longRunningBuildProject50> and a collection containing <P/longRunningBuildProject51> and a collection containing <P/longRunningBuildProject52> and a collection containing <P/longRunningBuildProject53> and a collection containing <P/longRunningBuildProject54> and a collection containing <P/longRunningBuildProject55> and a collection containing <P/longRunningBuildProject56> and a collection containing <P/longRunningBuildProject57> and a collection containing <P/longRunningBuildProject58> and a collection containing <P/longRunningBuildProject59> and a collection containing <P/longRunningBuildProject6> and a collection containing <P/longRunningBuildProject7> and a collection containing <P/longRunningBuildProject8> and a collection containing <P/longRunningBuildProject9>)
but: a collection containing <P/longRunningBuildProject10> mismatches were: [was <P/longRunningBuildProject0>, was <P/longRunningBuildProject12>, was <P/longRunningBuildProject22>, was <P/longRunningBuildProject34>, was <P/longRunningBuildProject40>, was <P/longRunningBuildProject50>, was <P/longRunningBuildProject37>, was <P/longRunningBuildProject59>, was <P/longRunningBuildProject16>, was <P/longRunningBuildProject57>, was <P/longRunningBuildProject39>, was <P/longRunningBuildProject47>, was <P/longRunningBuildProject7>, was <P/longRunningBuildProject13>, was <P/longRunningBuildProject24>, was <P/longRunningBuildProject2>, was <P/longRunningBuildProject19>, was <P/longRunningBuildProject8>, was <P/longRunningBuildProject55>, was <P/longRunningBuildProject6>, was <P/longRunningBuildProject18>, was <P/longRunningBuildProject42>, was <P/longRunningBuildProject58>, was <P/longRunningBuildProject52>, was <P/longRunningBuildProject33>, was <P/longRunningBuildProject54>, was <P/longRunningBuildProject17>, was <P/longRunningBuildProject30>, was <P/longRunningBuildProject38>, was <P/longRunningBuildProject56>, was <P/longRunningBuildProject23>, was <P/longRunningBuildProject41>, was <P/longRunningBuildProject25>, was <P/longRunningBuildProject31>, was <P/longRunningBuildProject45>, was <P/longRunningBuildProject14>, was <P/longRunningBuildProject15>, was <P/longRunningBuildProject35>, was <P/longRunningBuildProject32>, was <P/longRunningBuildProject1>, was <P/longRunningBuildProject48>, was <P/longRunningBuildProject3>, was <P/longRunningBuildProject27>, was <P/longRunningBuildProject53>, was <P/longRunningBuildProject44>, was <P/longRunningBuildProject49>, was <P/longRunningBuildProject29>, was <P/longRunningBuildProject4>, was <P/longRunningBuildProject28>, was <P/longRunningBuildProject11>, was <P/longRunningBuildProject20>]
at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:20)
at org.eclipse.core.tests.internal.builders.ParallelBuildChainTest.assertBuildsToStart(ParallelBuildChainTest.java:483)
at org.eclipse.core.tests.internal.builders.ParallelBuildChainTest.lambda$2(ParallelBuildChainTest.java:150)
at org.eclipse.core.tests.internal.builders.ParallelBuildChainTest.executeIndividualFullProjectBuilds(ParallelBuildChainTest.java:455)
at org.eclipse.core.tests.internal.builders.ParallelBuildChainTest.testIndividualProjectBuilds_WithManyProjects_ProjectRelaxedRule(ParallelBuildChainTest.java:149)

Nov 06 '23 07:11 HeikoKlare

Yes, I assume our build infrastructure is very unstable / slow on Linux. So if the test can increase timeout, it would be better as to disable this test completely.

Nov 06 '23 08:11 iloveeclipse

I increased timeout in #832. Let's see whether that helps. If it does not solve the problem, I will have a deeper look and probably adapt the test, as a test running for more than 30 seconds does not really make sense (except when being a performance test).

Nov 06 '23 11:11 HeikoKlare

Still randomly fails with timeout increase from 20 seconds to 30 seconds: https://download.eclipse.org/eclipse/downloads/drops4/I20231107-1800/testresults/html/org.eclipse.core.tests.resources_ep430I-unit-cen64-gtk3-java21_linux.gtk.x86_64_21.html

I will have a look whether that's really only an issue of performance or whether something is functionally broken.

Nov 08 '23 07:11 HeikoKlare

failed again for https://github.com/eclipse-platform/eclipse.platform/pull/1068

Jan 04 '24 08:01 BeckerWdf

In I-Builds the test now fails not because the because not all builds are started but eben because not all jobs can be spawned, see https://ci.eclipse.org/platform/job/eclipse.platform/job/PR-1386/2/testReport/

Timeout after 10000ms waiting for status to change from WAIT_FOR_START to RUNNING
 [ThreadDump taken from thread 'main' at 2024-05-21 10:15:32.062:
   Thread "main" #1 prio=5 RUNNABLE
     at [email protected]/java.lang.Thread.dumpThreads(Native Method)
     at [email protected]/java.lang.Thread.getAllStackTraces(Thread.java:1662)
     at org.eclipse.core.tests.harness.TestBarrier2.getThreadDump(TestBarrier2.java:109)
     at org.eclipse.core.tests.harness.TestBarrier2.doWaitForStatus(TestBarrier2.java:93)
     at org.eclipse.core.tests.harness.TestBarrier2.waitForStatus(TestBarrier2.java:160)
     at org.eclipse.core.tests.harness.TestBarrier2.waitForStatus(TestBarrier2.java:198)
     at org.eclipse.core.tests.internal.builders.ParallelBuildChainTest.executeIndividualFullProjectBuilds(ParallelBuildChainTest.java:453)
     at org.eclipse.core.tests.internal.builders.ParallelBuildChainTest.testIndividualProjectBuilds_WithManyProjects_ProjectRelaxedRule(ParallelBuildChainTest.java:150)

May 21 '24 13:05 HeikoKlare

still randomly failing:

Timeout after 10030ms waiting for status to change from WAIT_FOR_START to RUNNING
 [ThreadDump taken from thread 'main' at 2024-06-06 09:02:07.932:
   Thread "main" #1 prio=5 RUNNABLE
     at [email protected]/java.lang.Thread.dumpThreads(Native Method)
     at [email protected]/java.lang.Thread.getAllStackTraces(Thread.java:1662)
     at org.eclipse.core.tests.harness.TestBarrier2.getThreadDump(TestBarrier2.java:109)
     at org.eclipse.core.tests.harness.TestBarrier2.doWaitForStatus(TestBarrier2.java:93)
     at org.eclipse.core.tests.harness.TestBarrier2.waitForStatus(TestBarrier2.java:160)
     at org.eclipse.core.tests.harness.TestBarrier2.waitForStatus(TestBarrier2.java:198)
     at org.eclipse.core.tests.internal.builders.ParallelBuildChainTest.executeIndividualFullProjectBuilds(ParallelBuildChainTest.java:453)
     at org.eclipse.core.tests.internal.builders.ParallelBuildChainTest.testIndividualProjectBuilds_WithManyProjects_ProjectRelaxedRule(ParallelBuildChainTest.java:150)

https://ci.eclipse.org/platform/job/eclipse.platform/job/PR-1399/3/testReport/junit/(root)/AutomatedResourceTests%20AllBuilderTests%20ParallelBuildChainTest/testIndividualProjectBuilds_WithManyProjects_ProjectRelaxedRule/

Jun 06 '24 09:06 jukzi

happend again: https://download.eclipse.org/eclipse/downloads/drops4/I20241211-1800/testresults/html/org.eclipse.core.tests.resources_ep435I-unit-linux-x86_64-java23_linux.gtk.x86_64_23.html

Dec 12 '24 08:12 jukzi

Random failing ParallelBuildChainTest.testIndividualProjectBuilds_WithManyProjects_ProjectRelaxedRule