Patch filter test - Issue #1253

[sofiabobadilla]

Change GitHub.connectAnonymously() in PatchFilterTest.java to connect with Github using Jenkins environment variables instead.

[monperrus]

Great, CI seems to fail because of https://github.com/eclipse/repairnator/issues/1254#issuecomment-1231280182

[monperrus]

CI still failing. @javierron does Jenkins already know about credential GITHUB_OAUTH?

[javierron]

@monperrus @Sofi1410

Credentials used in the test class seem to be fine.

The problem is that the calls from inside the tested class don't use the GitHub client library and are probably being rate-limited (unclear from logs). We did it that way to be able "to change target URLs to a mirror in case rate limits are exceeded".

In case rate-limits are actually the issue, we would need to either use mirrors, or the GitHub client w/ token.

[monperrus]

Thanks for the analysis.

So we have to make readFromURL configurable (with or without token), and use the configurable version from PatchFilterTest.

WDYT?

[sofiabobadilla]

The CI keeps falling with this error: Caused by: java.lang.ClassNotFoundException: org.codehaus.plexus.util.xml.Xpp3DomBuilder$InputLocationBuilder

Apparently, this is a common bug related to dependencies. I will try this solution :https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=948286

[monperrus]

Yes, that's known as the "dependency hell", a topic we will research about in project Chains.

[sofiabobadilla]

FYI: Some tests were failing because they were using SpoonLab/sorald repository to check the runs, but the bot that was supposed to fail (codecove) is presenting some conflict therefore I change the repo to castor-software/depclean it fulfills the requirements to be a Maven project, public and with a CI with both successful and failed runs

[monperrus]

ack. cool to see progress here.

[sofiabobadilla]

@andre15silva here is the link of the test related to the dependency error we discuss at lunch: https://ci.eclipse.org/repairnator/job/repairnator-ci/job/PR-1256/44/

[andre15silva]

Hi @Sofi1410 ,

Seems like both errors come from flacoco executions:

java.lang.RuntimeException: java.nio.file.NoSuchFileException: /home/jenkins/agent/workspace/repairnator-ci_PR-1256/src/maven-repair/./target/CoveredTestResultPerTest.dat
	at eu.stamp_project.testrunner.listener.utils.ListenerUtils.loadFromMemoryMappedFile(ListenerUtils.java:82)
	at eu.stamp_project.testrunner.listener.impl.OnlineCoveredTestResultPerTestMethodImpl.load(OnlineCoveredTestResultPerTestMethodImpl.java:169)
	at eu.stamp_project.testrunner.EntryPoint.runOnlineCoveredTestResultPerTestMethods(EntryPoint.java:664)
	at fr.spoonlabs.flacoco.core.coverage.framework.JUnit4Strategy.execute(JUnit4Strategy.java:27)
	at fr.spoonlabs.flacoco.core.coverage.CoverageRunner.getCoverageMatrix(CoverageRunner.java:49)
	at fr.spoonlabs.flacoco.localization.spectrum.SpectrumRunner.computeCoverageMatrix(SpectrumRunner.java:56)
	at fr.spoonlabs.flacoco.localization.spectrum.SpectrumRunner.run(SpectrumRunner.java:32)
	at fr.spoonlabs.flacoco.api.Flacoco.run(Flacoco.java:31)

IIRC, this error usually appears due to RAM related issues (OOM). See https://github.com/ASSERT-KTH/flacoco/issues/121

From what I see, the amount of RAM available for each build is 8GiB. However, this may not be the amount of memory available to the JVM so it could be the case that it is indeed running out of memory. This would also answer the question of why you are not able to reproduce this error locally.

[andre15silva]

Btw @Sofi1410, if we identify a set of flaky tests we could possibly ignore them if they are not super important. Do you have this set?

@monperrus agrees that having CI back is more important

[monperrus]

yes, very much, we can delete flaky tests (better than @ignore).

[monperrus]

also FTR, we discussed with @martinwitt to have a github actions, only for compiling for now (mvn test -DskipTests, also compiles the tests)

[sofiabobadilla]

Great, I'll consider this for the current failing problems.

[MartinWitt]

Hey, I'm kind of out of the loop here, do you need any help?

[sofiabobadilla]

Hey, I'm kind of out of the loop here, do you need any help?

Do you mean for the GitHUb Actions? Anyways, might be better if we both sit and talk, do you have time tomorrow? :) @MartinWitt

[MartinWitt]

Sry I haven't read your comment till now. I'm in the office normally. Feel free to come over.

[MartinWitt]

Hey Sofia, junit-4 @Ignore has a field Reason with String type. You could add your fixme comments inside the field. See https://junit.org/junit4/javadoc/4.13.2/org/junit/Ignore.html#value()

[sofiabobadilla]

Thanks for the suggestion :)

[sofiabobadilla]

@algomaster99 I'm getting the sorald plugin error on CI again. Could you share the experiments you did last time when you checked the bug?

So far the error keps being:

• Could not download Sonar Java plugin -Caused by: java.nio.file.NoSuchFileException: /home/jenkins/.cache/sorald/sonar-java-plugin.jar

I double check the URL in sorald (https://github.com/ASSERT-KTH/sorald/blob/380e8158e56dc5464f9e750231b47bbc43b9d94a/sorald/src/main/resources/config.properties#L1) and it is working.

[sofiabobadilla]

At the same time I'm not entirely sure if it is related to the error I comment above, but, on local, the test fr.inria.spirals.repairnator.process.inspectors.TestProjectInspector4Maven.testPatchFailingProject reproduces the same error as CI only if I don't provide the M2_HOME var, but this parameter is present o the Jenkins File so I still don't understand completely what is going on.

[algomaster99]

Could you share the experiments you did last time when you checked the bug?

I did my experiment here. Looking back at it, I did not do anything because the problem did not persist.

So far the error keps being:

Could not download Sonar Java plugin -Caused by: java.nio.file.NoSuchFileException: /home/jenkins/.cache/sorald/sonar-java-plugin.jar

I double check the URL in sorald (https://github.com/ASSERT-KTH/sorald/blob/380e8158e56dc5464f9e750231b47bbc43b9d94a/sorald/src/main/resources/config.properties#L1) and it is working.

Successfully reaching this URL is flaky. Sometimes, it is fetched successfully, sometimes the CI just fails. I would recommend you to run you CI job again and it should pass.

[sofiabobadilla]

Could you share the experiments you did last time when you checked the bug?

I did my experiment here. Looking back at it, I did not do anything because the problem did not persist.

I'm looking at the execution: https://ci.eclipse.org/repairnator/job/repairnator-ci/job/PR-1263/2/testReport/ And there is no info related to sorald, the test report doesn't shows result from TestPipelineSoraldRepairTool. From my POV I don't see how the problem is solved.

Successfully reaching this URL is flaky. Sometimes, it is fetched successfully, sometimes the CI just fails. I would recommend you to run you CI job again and it should pass.

I don't understand how reaching the URL could be flaky for this, could you extend a bit on this?

[algomaster99]

Now, when I look back at the report, it seems that the test pipeline timed out and it never really executed the Sorald tests. I was trying to minimise the tests run in the build and accidentally prevented execution of the Sorald tests. I cannot test it right now because I am away from keyboard.

However, the errors which you are posting it is because of failure to fetch the jar at runtime.

I don't understand how reaching the URL could be flaky for this, could you extend a bit on this?

It is not flaky for this. It is flaky in general. Sometimes the CI is just not able to reach the URL due to network issues. It happens in Sorald's master branch as well, but it is quite rare (1 in 20 builds). The best thing to do here is to manually run it again. I also do it and 99% of the times the problem disappears if it could not fetch the jar in the previous build. Or you could have a check in the CI to run Sorald's test at least 2-3 times so that at least once you fetch the Jar.

[sofiabobadilla]

Related to last commit: https://bugs.eclipse.org/bugs/show_bug.cgi?id=575400