[Bug]: Intermittent test failure in master for :jballerina-debugger-integration-test:test

[lochana-chathura]

Description

$subject. Noticed :jballerina-debugger-integration-test:test failure in master quite frequently in recent times. Has to trigger the builds mutiple times to get the build passed.

Sample: https://github.com/ballerina-platform/ballerina-lang/actions/runs/6859271233/job/18651316246?pr=41683

debugger-intg-test-suite > debugger-intg-tests > org.ballerinalang.debugger.test.adapter.run.MultiModuleRunDebugTest > testMultiModuleDebugScenarios FAILED
    org.ballerinalang.test.context.BallerinaTestException: Breakpoints request failed.
        at app//org.ballerinalang.debugger.test.utils.DebugTestRunner.setBreakpoints(DebugTestRunner.java:298)
        at app//org.ballerinalang.debugger.test.utils.DebugTestRunner.addBreakPoint(DebugTestRunner.java:272)
        at app//org.ballerinalang.debugger.test.adapter.run.MultiModuleRunDebugTest.testMultiModuleDebugScenarios(MultiModuleRunDebugTest.java:109)

        Caused by:
        java.util.concurrent.TimeoutException
Nov 14, 2023 5:19:59 AM org.eclipse.lsp4j.jsonrpc.RemoteEndpoint fallbackResponseError
SEVERE: Internal error: com.sun.jdi.VMDisconnectedException: connection is closed
java.util.concurrent.CompletionException: com.sun.jdi.VMDisconnectedException: connection is closed
	at java.base/java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:315)
	at java.base/java.util.concurrent.CompletableFuture.completeThrowable(CompletableFuture.java:320)
	at java.base/java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1770)
	at java.base/java.util.concurrent.CompletableFuture$AsyncSupply.exec(CompletableFuture.java:1760)
	at java.base/java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:373)
	at java.base/java.util.concurrent.ForkJoinPool$WorkQueue.topLevelExec(ForkJoinPool.java:1182)
	at java.base/java.util.concurrent.ForkJoinPool.scan(ForkJoinPool.java:1655)
	at java.base/java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1622)
	at java.base/java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:165)
Caused by: com.sun.jdi.VMDisconnectedException: connection is closed
	at jdk.jdi/com.sun.tools.jdi.TargetVM.send(TargetVM.java:299)
	at jdk.jdi/com.sun.tools.jdi.VirtualMachineImpl.sendToTarget(VirtualMachineImpl.java:1169)
	at jdk.jdi/com.sun.tools.jdi.PacketStream.send(PacketStream.java:77)

 debugger-intg-test-suite > debugger-intg-tests > org.ballerinalang.debugger.test.adapter.BreakpointVerificationTest > testOnTheFlyBreakpointVerification FAILED
    org.ballerinalang.test.context.BallerinaTestException: Breakpoints request failed.
        at app//org.ballerinalang.debugger.test.utils.DebugTestRunner.setBreakpoints(DebugTestRunner.java:298)
        at app//org.ballerinalang.debugger.test.utils.DebugTestRunner.addBreakPoint(DebugTestRunner.java:272)
        at app//org.ballerinalang.debugger.test.adapter.BreakpointVerificationTest.testOnTheFlyBreakpointVerification(BreakpointVerificationTest.java:102)

        Caused by:
        java.util.concurrent.TimeoutException
            at java.base/java.util.concurrent.CompletableFuture.timedGet(CompletableFuture.java:1960)
            at java.base/java.util.concurrent.CompletableFuture.get(CompletableFuture.java:2095)
            at org.ballerinalang.debugger.test.utils.client.DAPRequestManager.setBreakpoints(DAPRequestManager.java:90)
            at org.ballerinalang.debugger.test.utils.client.DAPRequestManager.setBreakpoints(DAPRequestManager.java:84)
            at org.ballerinalang.debugger.test.utils.DebugTestRunner.setBreakpoints(DebugTestRunner.java:295)
            ... 2 more

 debugger-intg-test-suite > debugger-intg-tests > org.ballerinalang.debugger.test.adapter.completions.DebugCompletionTest > testDebugCompletions FAILED
    org.ballerinalang.test.context.BallerinaTestException: Error occurred when fetching completions
        at app//org.ballerinalang.debugger.test.utils.DebugTestRunner.fetchCompletions(DebugTestRunner.java:822)
        at app//org.ballerinalang.debugger.test.adapter.completions.DebugCompletionTest.testDebugCompletions(DebugCompletionTest.java:64)

        Caused by:
        java.util.concurrent.TimeoutException
            at java.base/java.util.concurrent.CompletableFuture.timedGet(CompletableFuture.java:1960)
            at java.base/java.util.concurrent.CompletableFuture.get(CompletableFuture.java:2095)
            at org.ballerinalang.debugger.test.utils.client.DAPRequestManager.completions(DAPRequestManager.java:155)
            at org.ballerinalang.debugger.test.utils.client.DAPRequestManager.completions(DAPRequestManager.java:149)
            at org.ballerinalang.debugger.test.utils.DebugTestRunner.fetchCompletions(DebugTestRunner.java:817)
            ... 1 more

[lochana-chathura]

https://github.com/ballerina-platform/ballerina-lang/actions/runs/6861344997/job/18656859525?pr=41693 build resulted in 178 tests completed, 8 failed, 116 skipped for the debugger tests

[NipunaRanasinghe]

The aforementioned failures are not reproducible locally and, there are no memory leaks found in the test suite. Therefore most probably this can be a cleanup glitch in the previous build/test tasks, leaving GitHub runner VMs short on resources to run the integration tests. (Noticed similar failures in LS perf tests too, probably due to the same reason).

Therefore to address these issues and to reduce our PR build time, we're planning to move all the integration tests (debugger, Testerina, and JBalllerina) into a separate job, and to run them in parallel.

In the meantime we will disable the most frequently failing test cases (mentioned in the issue description) ftm, and will enable them once the test parallelization work is done.