openj9 icon indicating copy to clipboard operation
openj9 copied to clipboard
verified publisher icon eclipse-openj9

revisit disabled tests

Open llxia opened this issue 5 years ago • 29 comments

We should revisit disabled tests to see if the issues are resolved. If they are, we need to re-enable the tests. For these ones that do not have any comments, we should check them again.

For example: https://github.com/eclipse/openj9/blob/ad7c3f09279a54f8d4d253af03da2a2a31f86f8e/test/functional/cmdLineTests/modularityddrtests/playlist.xml#L26 https://github.com/eclipse/openj9/blob/e2c38649abb06128ee30c06691b6dcc9e19db55b/test/functional/VM_Test/playlist.xml#L148 https://github.com/eclipse/openj9/blob/ab7f03ab6faa069307a3ddedf09ac26f79b9702c/test/functional/cmdLineTests/pltest/playlist.xml#L175

llxia avatar Oct 16 '20 21:10 llxia

@KapilPowar could you take a look at this issue? Thanks

llxia avatar Jun 12 '23 13:06 llxia

@pshipton

While testing VM test suite for JDK8 we are seeing test case failures. https://github.com/eclipse/openj9/blob/e2c38649abb06128ee30c06691b6dcc9e19db55b/test/functional/VM_Test/playlist.xml#L148

Here is the grinder link

Test cases enabled -- recreateClassTest (1/20 failed ) CFdumptest (1/8 failed)

There was 1 failure:
17:13:10  1) testInnerClasses(com.ibm.j9.recreateclass.tests.RecreateClassCompareTest)junit.framework.AssertionFailedError: Incorrect number of mismatches found expected:<1> but was:<5>
17:13:10  	at com.ibm.j9.recreateclass.tests.RecreateClassCompareTest.compareClassFiles(RecreateClassCompareTest.java:63)
17:13:10  	at com.ibm.j9.recreateclass.tests.RecreateClassCompareTest.runCfdumpAndCompareTools(RecreateClassCompareTest.java:74)
17:13:10  	at com.ibm.j9.recreateclass.tests.RecreateClassCompareTest.runTest(RecreateClassCompareTest.java:86)
17:13:10  	at com.ibm.j9.recreateclass.tests.RecreateClassCompareTest.runTest(RecreateClassCompareTest.java:79)
17:13:10  	at com.ibm.j9.recreateclass.tests.RecreateClassCompareTest.testInnerClasses(RecreateClassCompareTest.java:154)
17:13:10  	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
17:13:10  	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
17:13:10  	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
17:13:10  
17:13:10  FAILURES!!!
17:13:10  Tests run: 20,  Failures: 1,  Errors: 0
--------------------------------------------------------------------

CFdumptest (1/8 failed)

There was 1 failure:
17:13:11  1) testInvalidClassFile(com.ibm.j9.cfdump.tests.lineardump.LinearDumpTests)junit.framework.AssertionFailedError: Expected: <null> but was: bad magic number
17:13:11  	at com.ibm.j9.cfdump.tests.lineardump.LinearDumpTests.testInvalidClassFile(LinearDumpTests.java:123)
17:13:11  	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
17:13:11  	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
17:13:11  	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
17:13:11  
17:13:11  FAILURES!!!
17:13:11  Tests run: 8,  Failures: 1,  Errors: 0

Please confirm if it is product issue.

KapilPowar avatar Jul 06 '23 12:07 KapilPowar

How do I re-run these tests? The grinder link is expired. When I try to run them every test fails with Error: cfdump not found on java.library.path!. I see that cfdump is part of the test image.

https://openj9-jenkins.osuosl.org/view/Test/job/Grinder/2649

pshipton avatar Aug 04 '23 16:08 pshipton

In playlist.xml file please try to add cfdump in the lib path - using $(ADD_JVM_LIB_DIR_TO_LIBPATH) \ in <command> for the test cases.

e.g. <command>$(ADD_JVM_LIB_DIR_TO_LIBPATH) \ $(JAVA_COMMAND) $(JVM_OPTIONS) \

KapilPowar avatar Aug 07 '23 09:08 KapilPowar

Here are latest Grinder Links aarch64_linux --Java 8 x86-64_linux -- Java 8 x86-64_windows -- Java 8

KapilPowar avatar Aug 09 '23 05:08 KapilPowar

Failure details from recreateClassTest

13:15:46  Start testInnerClasses
13:15:46  Running command: /home/jenkins/workspace/Grinder/openjdkbinary/openjdk-test-image/openj9/cfdump -t:/home/jenkins/workspace/Grinder/aqa-tests/TKG/output_16917741452730/recreateClassTest_0/InnerClassesTest /home/jenkins/workspace/Grinder/aqa-tests/TKG/output_16917741452730/recreateClassTest_0/InnerClassesTest.class
13:15:46  cfdump: Wrote 1212 bytes to output file /home/jenkins/workspace/Grinder/aqa-tests/TKG/output_16917741452730/recreateClassTest_0/InnerClassesTest.j9class
13:15:46  Inner class access flags differ: 0X8, 0X0
13:15:46  Aug 11, 2023 5:15:46 PM com.ibm.j9.test.tools.classfilecompare.ClassFileCompare reportDifference
13:15:46  WARNING: Inner class access flags differ: 0X8, 0X0
13:15:46  Inner class inner name differ: LocalClass, null
13:15:46  Aug 11, 2023 5:15:46 PM com.ibm.j9.test.tools.classfilecompare.ClassFileCompare reportDifference
13:15:46  WARNING: Inner class inner name differ: LocalClass, null
13:15:46  Inner class access flags differ: 0X8, 0X0
13:15:46  Aug 11, 2023 5:15:46 PM com.ibm.j9.test.tools.classfilecompare.ClassFileCompare reportDifference
13:15:46  WARNING: Inner class access flags differ: 0X8, 0X0
13:15:46  Inner class access flags differ: 0X9, 0X0
13:15:46  Aug 11, 2023 5:15:46 PM com.ibm.j9.test.tools.classfilecompare.ClassFileCompare reportDifference
13:15:46  WARNING: Inner class access flags differ: 0X9, 0X0
13:15:46  Inner class inner name differ: InnerClass, null
13:15:46  Aug 11, 2023 5:15:46 PM com.ibm.j9.test.tools.classfilecompare.ClassFileCompare reportDifference
13:15:46  WARNING: Inner class inner name differ: InnerClass, null
13:15:46  Mismatches spotted: 5

pshipton avatar Aug 11 '23 17:08 pshipton

@TobiAjila can someone pls take a look at this recreateClassTest failure in the previous comment, it appears we may need a fix.

My changes to run the tests and add debug are in https://github.com/pshipton/openj9/tree/tests

pshipton avatar Aug 11 '23 17:08 pshipton

@hangshao0 Can you please take a look at this

tajila avatar Aug 11 '23 18:08 tajila

Re-enabling CFdumptest via https://github.com/eclipse-openj9/openj9/pull/17939

pshipton avatar Aug 11 '23 20:08 pshipton

I will look at this.

hangshao0 avatar Aug 14 '23 14:08 hangshao0

I've manually run cfdump on the inner class. Looking at the generated j9class and original class file:

cfdump -d 'InnerClassesTest$InnerClass.class'
Magic: CAFEBABE
Version: 52.0
Constant Pool Size: 17
Access Flags: 0x21 (
  access: public super )
ThisClass: 2 -> InnerClassesTest$InnerClass
SuperClass: 3 -> java/lang/Object
Interfaces (0):

Fields (0):
Methods (1):
  Name: 4 -> <init>
  Signature: 5 -> ()V
  Access Flags: 0x1 (
  access: public )
  Attributes (1):
    Code:
      Max stack: 1
      Max Locals: 1
      Code Length: 5
      Caught Exceptions (0):
      Attributes (1):
        LineNumberTable:
          pc: 0 line number: 2

Attributes (2):
  SourceFile:
    Source File: 9 -> InnerClassesTest.java
  InnerClasses:
    Inner Class: 2 -> InnerClassesTest$InnerClass
      Outer Class: 11 -> InnerClassesTest
      Inner Name: 13 -> InnerClass
      Inner Class Access Flags: 0x9 (
  access: public )

cfdump -d 'InnerClassesTest$InnerClass.j9class'
Magic: CAFEBABE
Version: 52.0
Constant Pool Size: 17
Access Flags: 0x21 (
  access: public super )
ThisClass: 5 -> InnerClassesTest$InnerClass
SuperClass: 2 -> java/lang/Object
Interfaces (0):

Fields (0):
Methods (1):
  Name: 15 -> <init>
  Signature: 3 -> ()V
  Access Flags: 0x1 (
  access: public )
  Attributes (1):
    Code:
      Max stack: 1
      Max Locals: 1
      Code Length: 5
      Caught Exceptions (0):
      Attributes (1):
        LineNumberTable:
          pc: 0 line number: 2

Attributes (2):
  InnerClasses:
    Inner Class: 5 -> InnerClassesTest$InnerClass
      Outer Class: 12 -> InnerClassesTest
      Inner Name: 8 -> InnerClass
      Inner Class Access Flags: 0x9 (
  access: public )
  SourceFile:
    Source File: 13 -> InnerClassesTest.java

I see the access flags and inner class names are the same. The differences might be from the outer class that cause the test to fail.

hangshao0 avatar Aug 15 '23 20:08 hangshao0 

cfdump -d InnerClassesTest.class
Magic: CAFEBABE
Version: 52.0
Constant Pool Size: 24
Access Flags: 0x21 (
  access: public super )
ThisClass: 3 -> InnerClassesTest
SuperClass: 4 -> java/lang/Object
Interfaces (0):

Fields (0):
Methods (2):
  Name: 9 -> <init>
  Signature: 10 -> ()V
  Access Flags: 0x1 (
  access: public )
  Attributes (1):
    Code:
      Max stack: 1
      Max Locals: 1
      Code Length: 5
      Caught Exceptions (0):
      Attributes (1):
        LineNumberTable:
          pc: 0 line number: 1

  Name: 13 -> getLocalClass
  Signature: 14 -> ()Ljava/lang/Class;
  Access Flags: 0x9 (
  access: public static )
  Attributes (2):
    Code:
      Max stack: 1
      Max Locals: 0
      Code Length: 3
      Caught Exceptions (0):
      Attributes (1):
        LineNumberTable:
          pc: 0 line number: 8
    Signature:
      Signature: 16 -> ()Ljava/lang/Class<*>;

Attributes (2):
  SourceFile:
    Source File: 18 -> InnerClassesTest.java
  InnerClasses:
    Inner Class: 5 -> InnerClassesTest$InnerClass
      Outer Class: 3 -> InnerClassesTest
      Inner Name: 6 -> InnerClass
      Inner Class Access Flags: 0x9 (
  access: public )
    Inner Class: 2 -> InnerClassesTest$1LocalClass
      Outer Class: (not a member)
      Inner Name: 8 -> LocalClass
      Inner Class Access Flags: 0x0 (
  access: default )

cfdump -d InnerClassesTest.j9class
Magic: CAFEBABE
Version: 52.0
Constant Pool Size: 22
Access Flags: 0x21 (
  access: public super )
ThisClass: 7 -> InnerClassesTest
SuperClass: 3 -> java/lang/Object
Interfaces (0):

Fields (0):
Methods (2):
  Name: 21 -> <init>
  Signature: 14 -> ()V
  Access Flags: 0x1 (
  access: public )
  Attributes (1):
    Code:
      Max stack: 1
      Max Locals: 1
      Code Length: 5
      Caught Exceptions (0):
      Attributes (1):
        LineNumberTable:
          pc: 0 line number: 1

  Name: 12 -> getLocalClass
  Signature: 13 -> ()Ljava/lang/Class;
  Access Flags: 0x9 (
  access: public static )
  Attributes (2):
    Code:
      Max stack: 1
      Max Locals: 0
      Code Length: 3
      Caught Exceptions (0):
      Attributes (1):
        LineNumberTable:
          pc: 0 line number: 8
    Signature:
      Signature: 4 -> ()Ljava/lang/Class<*>;

Attributes (2):
  InnerClasses:
    Inner Class: 20 -> InnerClassesTest$InnerClass
      Outer Class: 7 -> InnerClassesTest
      Inner Name: (anonymous)                 <------------------------------------- this is different from the original class
      Inner Class Access Flags: 0x0 (         <------------------------------------- this is different from the original class
  access: default )
    Inner Class: 1 -> InnerClassesTest$1LocalClass
      Outer Class: (not a member)
      Inner Name: (anonymous)
      Inner Class Access Flags: 0x0 (
  access: default )
  SourceFile:
    Source File: 5 -> InnerClassesTest.java

The difference is from the inner class attribute of the outer class. The inner class simple name and access flags are not preserved in the romClass of the outer class: https://github.com/eclipse-openj9/openj9/blob/949a4df06ef763ed4f3a83faf8d9534f911f4d07/runtime/bcutil/ClassFileWriter.cpp#L1054-L1056

If we decided to not record these information in the romClass in #13199, the test needs to be updated to accept such differences in .class and .j9class files.

hangshao0 avatar Aug 16 '23 20:08 hangshao0

@ChengJin01 ^^

hangshao0 avatar Aug 16 '23 20:08 hangshao0

CFdumptest is only enabled for JDK8. I think it is able to run against all versions. https://github.com/pshipton/openj9/blob/master/test/functional/VM_Test/playlist.xml#L167 @KapilPowar could you run CFdumptest on JDK11 and JDK17 when you have time? Thanks

llxia avatar Aug 23 '23 20:08 llxia

I tried other versions but there are problems that would need to be resolved.

pshipton avatar Aug 24 '23 01:08 pshipton

Grinder test runs for CFdumptest JDK11 x86-64_linux JDK11 x86-64_mac JDK11 x86-64_windows JDK17 x86-64_linux JDK17 x86-64_mac JDK17 x86-64_windows

KapilPowar avatar Aug 29 '23 11:08 KapilPowar

Thanks @KapilPowar . Could also run on other missing platforms in Grinder? Thanks

llxia avatar Aug 29 '23 14:08 llxia

Grinder test runs for CFdumptest JDK17 s390x_linux --UNSTABLE aarch64_mac --Success ppc64_aix --UNSTABLE

JDK11 aarch64_mac -- Success s390x_linux--UNSTABLE ppc64_aix --UNSTABLE

KapilPowar avatar Aug 30 '23 15:08 KapilPowar

@llxia @pshipton FYI Grinder test run for recreateClassTest still results in same error -- JDK8 x86-64_linux

There was 1 failure:
17:01:36  1) testInnerClasses(com.ibm.j9.recreateclass.tests.RecreateClassCompareTest)junit.framework.AssertionFailedError: Incorrect number of mismatches found expected:<1> but was:<5>
17:01:36  	at com.ibm.j9.recreateclass.tests.RecreateClassCompareTest.compareClassFiles(RecreateClassCompareTest.java:63)
17:01:36  	at com.ibm.j9.recreateclass.tests.RecreateClassCompareTest.runCfdumpAndCompareTools(RecreateClassCompareTest.java:74)
17:01:36  	at com.ibm.j9.recreateclass.tests.RecreateClassCompareTest.runTest(RecreateClassCompareTest.java:86)
17:01:36  	at com.ibm.j9.recreateclass.tests.RecreateClassCompareTest.runTest(RecreateClassCompareTest.java:79)
17:01:36  	at com.ibm.j9.recreateclass.tests.RecreateClassCompareTest.testInnerClasses(RecreateClassCompareTest.java:154)
17:01:36  	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
17:01:36  	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
17:01:36  	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
17:01:36  
17:01:36  FAILURES!!!
17:01:36  Tests run: 20,  Failures: 1,  Errors: 0
17:01:36  
17:01:36  -----------------------------------
17:01:36  recreateClassTest_0_FAILED
17:01:36  -----------------------------------

KapilPowar avatar Sep 19 '23 11:09 KapilPowar

I assume this can be moved forward.

pshipton avatar Sep 19 '23 14:09 pshipton

Re https://github.com/eclipse-openj9/openj9/issues/10917#issuecomment-1725335609, we're waiting for @ChengJin01 to have time to take a look, based on https://github.com/eclipse-openj9/openj9/issues/10917#issuecomment-1681233241

pshipton avatar Sep 19 '23 14:09 pshipton

I agree with @hangshao0 's investigation & conclusions given we don' record the access flags in the romClass by default.

ChengJin01 avatar Sep 19 '23 14:09 ChengJin01

So the next step is to update the test itself to ignore those differences. @TobiAjila

pshipton avatar Sep 19 '23 14:09 pshipton

@llxia Grinder test run for cmdLineTester_forceLazySymbolResolution JDK17 x86-64_linux

Grinder test run for cmdLineTester_jvmtitests_hcr_OSRG_nongold JDK17 x86-64_linux JDK17 x86-64_mac JDK17 x86-64_windows

KapilPowar avatar Sep 21 '23 11:09 KapilPowar

@KapilPowar if the Grinders pass, please create PR to reenable the tests. Thanks

llxia avatar Sep 21 '23 13:09 llxia

@KapilPowar if the Grinders pass, please create PR to reenable the tests. Thanks

PR created https://github.com/eclipse-openj9/openj9/pull/18181

KapilPowar avatar Sep 22 '23 11:09 KapilPowar

@llxia Grinder test run for TestFlushReflectionCache & jit_jitt_openj9_none_SCC x86-64_linux x86-64_mac x86-64_windows

PR created #18206

KapilPowar avatar Sep 27 '23 13:09 KapilPowar

@llxia Grinder test run for cmdLineTester_dumpromclasstests aix_ppc-64 JDK8 aix_ppc-64 JDK11 aix_ppc-64 JDK17

PR created #18215

KapilPowar avatar Sep 29 '23 05:09 KapilPowar

@llxia Do we still need this issue?

tajila avatar May 14 '24 13:05 tajila

I believe this issue can be closed.

llxia avatar Aug 13 '24 13:08 llxia