repairnator icon indicating copy to clipboard operation
repairnator copied to clipboard

Remove or fix Flaky tests

Open andre15silva opened this issue 4 years ago • 4 comments

There are some flaky tests currently.

I have identified two so far:

  • https://github.com/eclipse/repairnator/blob/103e6415f012a58b4686a92c66b8d761f0f96253/src/repairnator-pipeline/src/test/java/fr/inria/spirals/repairnator/pipeline/TestPipeline.java#L55 Failed and Passed Reason: org.eclipse.jgit.api.errors.TransportException: Connection timed out (Connection timed out)

  • https://github.com/eclipse/repairnator/blob/103e6415f012a58b4686a92c66b8d761f0f96253/src/repairnator-pipeline/src/test/java/fr/inria/spirals/repairnator/pipelineb/TestPipelinebGithubMode.java#L67 Failed and Passed, Failed and Passed Can't determine reason because there's no access to test report

andre15silva avatar Jan 22 '21 12:01 andre15silva

Can we close this one?

monperrus avatar Mar 09 '21 06:03 monperrus

This still happens from time to time, so I think it's better to keep it open.

andre15silva avatar Mar 09 '21 11:03 andre15silva

Another flaky test identified:

  • https://github.com/eclipse/repairnator/blob/103e6415f012a58b4686a92c66b8d761f0f96253/src/repairnator-pipeline/src/test/java/fr/inria/spirals/repairnator/process/step/TestGatherTestInformation.java#L109 Failed and Passed Reason: sometimes the error message of the test project returns two lines without \n separating them

andre15silva avatar Mar 29 '21 17:03 andre15silva

TestMetrics4BearsJsonFile.testBearsJsonFileWithPassingPassingBuilds is flaky, last instance is https://ci.eclipse.org/repairnator/job/repairnator-ci/job/monperrus-patch-2/1/console

monperrus avatar Aug 09 '21 10:08 monperrus