eclipse-platform
[TEST][windows] Seeing new Junit failure from 16th Feb build onwards
Describe the bug test_setTextContainingScript_applicationLayerProgressListenerMustSeeUpToDateDom test case is failing.
Expected behavior This junit was passing in the previous builds, so did anything change recently regarding this?
Detailed error
Multiple Failures (2 failures) org.junit.ComparisonFailure: expected:<...ressListener: Found [1] h1 tag(s)> but was:<...ressListener: Found [0] h1 tag(s)> java.lang.RuntimeException: AwtScreenshot VM finished with exit code 1: Exception in thread "main" java.awt.AWTError: no screen devices at java.desktop/sun.awt.Win32GraphicsEnvironment.getDefaultScreenDevice(Win32GraphicsEnvironment.java:101) at java.desktop/java.awt.Robot.<init>(Robot.java:135) at org.eclipse.test.AwtScreenshot.main(AwtScreenshot.java:43)
org.opentest4j.MultipleFailuresError: Multiple Failures (2 failures)
org.junit.ComparisonFailure: expected:<...ressListener: Found [1] h1 tag(s)> but was:<...ressListener: Found [0] h1 tag(s)>
java.lang.RuntimeException: AwtScreenshot VM finished with exit code 1:
Exception in thread "main" java.awt.AWTError: no screen devices
at java.desktop/sun.awt.Win32GraphicsEnvironment.getDefaultScreenDevice(Win32GraphicsEnvironment.java:101)
at java.desktop/java.awt.Robot.<init>(Robot.java:135)
at org.eclipse.test.AwtScreenshot.main(AwtScreenshot.java:43)
at org.eclipse.test.TracingSuite$TracingRunNotifier.fireTestFinished(TracingSuite.java:176)
at org.eclipse.test.TracingSuite.runChild(TracingSuite.java:287)
Suppressed: org.junit.ComparisonFailure: expected:<...ressListener: Found [1] h1 tag(s)> but was:<...ressListener: Found [0] h1 tag(s)>
at org.junit.Assert.assertEquals(Assert.java:117)
at org.junit.Assert.assertEquals(Assert.java:146)
at org.eclipse.swt.tests.junit.Test_org_eclipse_swt_browser_Browser.test_setTextContainingScript_applicationLayerProgressListenerMustSeeUpToDateDom(Test_org_eclipse_swt_browser_Browser.java:1181)
at java.base/java.lang.reflect.Method.invoke(Method.java:580)
... 1 more
Suppressed: java.lang.RuntimeException: AwtScreenshot VM finished with exit code 1:
Exception in thread "main" java.awt.AWTError: no screen devices
at java.desktop/sun.awt.Win32GraphicsEnvironment.getDefaultScreenDevice(Win32GraphicsEnvironment.java:101)
at java.desktop/java.awt.Robot.<init>(Robot.java:135)
at org.eclipse.test.AwtScreenshot.main(AwtScreenshot.java:43)
at org.eclipse.test.AwtScreenshot.dumpAwtScreenshot(AwtScreenshot.java:113)
at org.eclipse.test.Screenshots.takeScreenshot(Screenshots.java:102)
at org.eclipse.test.Screenshots$ScreenshotOnFailure.failed(Screenshots.java:40)
... 1 more
Environment:
- Select the platform(s) on which the behavior is seen:
-
- [ ] All OS
-
- [x] Windows
-
- [ ] Linux
-
- [ ] macOS
-
Additional info Link to the doc -> https://download.eclipse.org/eclipse/downloads/drops4/I20250216-1800/testresults/html/org.eclipse.swt.tests_ep435I-unit-win32-x86_64-java21_win32.win32.x86_64_21.html
-
JRE/JDK version
Version since Seeing this junit failing is failing since "I20250216-1800" and on "I20250217-0600"
Workaround (or) Additional context Not aware of.
This passing on I20250218-0600
https://download.eclipse.org/eclipse/downloads/drops4/I20250218-0600/testResults.php
This is passing on I20250223-1800
https://download.eclipse.org/eclipse/downloads/drops4/I20250223-1800/testResults.php
So closing this issue as of now.
Again seeing the above failure -> https://download.eclipse.org/eclipse/downloads/drops4/I20250224-1800/testresults/html/org.eclipse.swt.tests_ep435I-unit-win32-x86_64-java21_win32.win32.x86_64_21.html
So opening this issue back.
@here : does anyone has any idea on what can cause this failure?
The failing test is a browser test: test_setTextContainingScript_applicationLayerProgressListenerMustSeeUpToDateDom[browser flags: 0]
The browser flag being 0 means that the test is executed without any flag, i.e., using Edge/WebView2.
The failing line of code is this: https://github.com/eclipse-platform/eclipse.platform.swt/blob/9f602795596c64003ea7b8c83e0153e5657a84f2/tests/org.eclipse.swt.tests/JUnit%20Tests/org/eclipse/swt/tests/junit/Test_org_eclipse_swt_browser_Browser.java#L1181
It indicates that the progress listener is called before the actual browser content has been set.
The failing test is a browser test:
test_setTextContainingScript_applicationLayerProgressListenerMustSeeUpToDateDom[browser flags: 0]The browser flag being 0 means that the test is executed without any flag, i.e., using Edge/WebView2.
Sorry i didnt understand it clearly. So do you mean it is expected failure? But in between it was passing? so is it a transient failure? or it can be fixed?
Sorry i didnt understand it clearly. So do you mean it is expected failure? But in between it was passing? so is it a transient failure? or it can be fixed?
No, sorry for being imprecise here. I don't think this is an expected failure. I just wanted to clarify what happens (i.e., what kind of failure the the failing test expresses), but one would have to analyze whether "only" the test is not properly implemented or whther something is wrong with the progress handling in the Edge implementation.
From the logs, we can at least see that it does not seem to be a timeout issue, as the failure occurs just one second after the test started (and timeout values are fairly higher):
[2025-02-24 23:51:32 +0000] org.eclipse.swt.tests.junit.Test_org_eclipse_swt_browser_Browser.test_setTextContainingScript_applicationLayerProgressListenerMustSeeUpToDateDom[browser flags: 0]()
Running Test_org_eclipse_swt_browser_Browser#test_setTextContainingScript_applicationLayerProgressListenerMustSeeUpToDateDom[browser flags: 0]
[2025-02-24 23:51:33 +0000] org.eclipse.swt.tests.junit.Test_org_eclipse_swt_browser_Browser.test_setTextContainingScript_applicationLayerProgressListenerMustSeeUpToDateDom[browser flags: 0]() failed:
org.junit.ComparisonFailure: expected:<...ressListener: Found [1] h1 tag(s)> but was:<...ressListener: Found [0] h1 tag(s)>
at org.junit.Assert.assertEquals(Assert.java:117)
at org.junit.Assert.assertEquals(Assert.java:146)
at org.eclipse.swt.tests.junit.Test_org_eclipse_swt_browser_Browser.test_setTextContainingScript_applicationLayerProgressListenerMustSeeUpToDateDom(Test_org_eclipse_swt_browser_Browser.java:1181)
Looks like the failure is related to the Windows/WebView2 version. The test used to fail only sporadically in GHA whereas it now also fails on Jenkins. I am able to reproduce the failure on system with Windows build 26100, but I cannot reproduce on a system with build 22631. Might also be related to the 26100 system being less performant than the older one. But might also be that Jenkins was updated to new Windows version and that introduced the failure there as well.