jakartaee
TCK challenge for adding support for running Security 3.0 signature TCK tests on Java 21
When running the https://www.eclipse.org/downloads/download.php?file=/jakartaee/security/3.0/jakarta-security-tck-3.0.1.zip tests on Java 21, the following signature test failures occur due to Java 21 added a few new JDK methods:
12-01-2023 11:32:57:********** Status Report 'jakarta.security.enterprise.identitystore.openid' ********** 12-01-2023 11:32:57:SignatureTest report Base version: 3.0.0 Tested version: 3.0.0 Check mode: src [throws normalized] Constant checking: on
Added Superclasses or Superinterfaces
jakarta.security.enterprise.identitystore.openid.Scope: interface java.util.SequencedCollection<java.lang.String> jakarta.security.enterprise.identitystore.openid.Scope: interface java.util.SequencedSet<java.lang.String>
Added Methods
jakarta.security.enterprise.identitystore.openid.Scope: method public java.lang.String java.util.LinkedHashSet.getFirst() jakarta.security.enterprise.identitystore.openid.Scope: method public java.lang.String java.util.LinkedHashSet.getLast() jakarta.security.enterprise.identitystore.openid.Scope: method public java.lang.String java.util.LinkedHashSet.removeFirst() jakarta.security.enterprise.identitystore.openid.Scope: method public java.lang.String java.util.LinkedHashSet.removeLast() jakarta.security.enterprise.identitystore.openid.Scope: method public java.util.SequencedSet<java.lang.String> java.util.LinkedHashSet.reversed() jakarta.security.enterprise.identitystore.openid.Scope: method public static <%0 extends java.lang.Object> java.util.HashSet<{%%0}> java.util.HashSet.newHashSet(int) jakarta.security.enterprise.identitystore.openid.Scope: method public static <%0 extends java.lang.Object> java.util.LinkedHashSet<{%%0}> java.util.LinkedHashSet.newLinkedHashSet(int) jakarta.security.enterprise.identitystore.openid.Scope: method public void java.util.LinkedHashSet.addFirst(java.lang.String) jakarta.security.enterprise.identitystore.openid.Scope: method public void java.util.LinkedHashSet.addLast(java.lang.String)
12-01-2023 11:32:57:********** Package 'jakarta.security.enterprise.identitystore.openid' - FAILED (STATIC MODE) **********
12-01-2023 11:32:58:********** Status Report 'jakarta.security.enterprise.identitystore.openid' **********
12-01-2023 11:32:58:SignatureTest report Base version: 3.0.0 Tested version: 3.0.0 Check mode: src [throws normalized] Constant checking: on
Added Superclasses or Superinterfaces
jakarta.security.enterprise.identitystore.openid.Scope: interface java.util.SequencedCollection<java.lang.String> jakarta.security.enterprise.identitystore.openid.Scope: interface java.util.SequencedSet<java.lang.String>
Added Methods
jakarta.security.enterprise.identitystore.openid.Scope: method public java.lang.String java.util.LinkedHashSet.getFirst() jakarta.security.enterprise.identitystore.openid.Scope: method public java.lang.String java.util.LinkedHashSet.getLast() jakarta.security.enterprise.identitystore.openid.Scope: method public java.lang.String java.util.LinkedHashSet.removeFirst() jakarta.security.enterprise.identitystore.openid.Scope: method public java.lang.String java.util.LinkedHashSet.removeLast() jakarta.security.enterprise.identitystore.openid.Scope: method public java.util.SequencedSet<java.lang.String> java.util.LinkedHashSet.reversed() jakarta.security.enterprise.identitystore.openid.Scope: method public static <%0 extends java.lang.Object> java.util.HashSet<{%%0}> java.util.HashSet.newHashSet(int) jakarta.security.enterprise.identitystore.openid.Scope: method public static <%0 extends java.lang.Object> java.util.LinkedHashSet<{%%0}> java.util.LinkedHashSet.newLinkedHashSet(int) jakarta.security.enterprise.identitystore.openid.Scope: method public void java.util.LinkedHashSet.addFirst(java.lang.String) jakarta.security.enterprise.identitystore.openid.Scope: method public void java.util.LinkedHashSet.addLast(java.lang.String)
12-01-2023 11:32:58:SigTestEE.signatureTest() failed!, diffs found [ERROR] Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 6.252 s <<< FAILURE! - in ee.jakarta.tck.security.signaturetest.SecurityAPISigTestIT [ERROR] ee.jakarta.tck.security.signaturetest.SecurityAPISigTestIT.signatureTest Time elapsed: 6.074 s <<< ERROR! ee.jakarta.tck.security.signaturetest.SigTestEE$Fault: SigTestEE.signatureTest() failed!, diffs found at [email protected]/ee.jakarta.tck.security.signaturetest.SecurityAPISigTestIT.signatureTest(SecurityAPISigTestIT.java:355)
There are two possible fixes, the easiest one line change would be to exclude the specific updated JDK classes on the command line when invoking the sigtest tool. I'll look into that next year.
The better change would be to update https://github.com/jtulach/netbeans-apitest to be able to avoid checking any SPEC API classes that are not in the jakarta namespace which I think would avoid validation of any JDK classes and instead focus only on checking jakarta.* classes. I'd like to do this change for Jakarta EE 11+.
Since all of the Signature test failures also happen in the Platform TCK signature tests we can probably address that first and likely make the same one line change in other places as well. I think we should turn these issues into TCK challenges so could one of the contributors please add the challenge label. Thanks!
I can confirm that the https://github.com/jakartaee/platform-tck/pull/1206 change solved ^ running the (Jakarta EE 10) Platform TCK Signature tests on Java 21 and the same change should help the Security TCK.
Since all of the Signature test failures also happen in the Platform TCK signature tests we can probably address that first and likely make the same one line change in other places as well. I think we should turn these issues into TCK challenges so could one of the contributors please add the
challengelabel. Thanks!
^ is done and the updated Platform TCK 10.0.3 TCK is released as mentioned on https://jakarta.ee/specifications/platform/10
Could one of the Security project committers please add the challenge label and initiate process to accept or reject this TCK challenge.
@scottmarlow sorry for the delay. Of course this will be done for the final release of the TCK. I just did the same for the Jakarta Authorization TCK.
See: https://github.com/jakartaee/authorization/tree/master/tck/authorization-signaturetest
@scottmarlow sorry for the delay. Of course this will be done for the final release of the TCK. I just did the same for the Jakarta Authorization TCK.
See: https://github.com/jakartaee/authorization/tree/master/tck/authorization-signaturetest
@arjantijms to be clear, I think that we need a https://www.eclipse.org/downloads/download.php?file=/jakartaee/security/3.0/jakarta-security-tck-3.0.2.zip release that includes the https://github.com/jakartaee/security/pull/305 fix as well.
I've staged 3.0.2 here, but not checked it yet:
https://www.eclipse.org/downloads/download.php?file=/security/jakartaee10/staged/eftl/jakarta-security-tck-3.0.2.zip
I've staged 3.0.2 here, but not checked it yet:
https://www.eclipse.org/downloads/download.php?file=/security/jakartaee10/staged/eftl/jakarta-security-tck-3.0.2.zip
Thanks, I'm working on running it locally with latest WildFly on Java 17 and will then run again on Java 21.
The JDK 21 run against GlassFish 7.0.15 succeeded: https://ci.eclipse.org/jakartaee-tck/view/EFTL-Certification-Jobs-10/job/10/job/jakarta-security-tck-glassfish/141/
I'm hitting environmental problems locally so probably a good idea to promote it as I did at least see the signature tests pass locally with JDK21:
All package signatures passed. Passed packages listed below: jakarta.security.enterprise(static mode) jakarta.security.enterprise(reflection mode) jakarta.security.enterprise.authentication.mechanism.http(static mode) jakarta.security.enterprise.authentication.mechanism.http(reflection mode) jakarta.security.enterprise.authentication.mechanism.http.openid(static mode) jakarta.security.enterprise.authentication.mechanism.http.openid(reflection mode) jakarta.security.enterprise.credential(static mode) jakarta.security.enterprise.credential(reflection mode) jakarta.security.enterprise.identitystore(static mode) jakarta.security.enterprise.identitystore(reflection mode) jakarta.security.enterprise.identitystore.openid(static mode) jakarta.security.enterprise.identitystore.openid(reflection mode)
@arjantijms any idea why Apache Tomcat might be used for some tests by default? I previously tested with the 3.0.0 TCK but not 3.0.1 (in case that helps). https://gist.github.com/scottmarlow/50802d6a60ca5228fa92d207b4d44a78 shows some evidence of org.apache.catalina.loader.WebappClassLoaderBase being used in my 3.0.2 TCK testing which sounds config related.
I also saw the first 3.0.2 TCK test failure with error ee.jakarta.tck.security.test.OpenId2DefaultIT which seems to be caused by Failed to start service org.wildfly.undertow.listener.https: org.jboss.msc.service.StartException in service org.wildfly.undertow.listener.https: Address already in use /127.0.0.1:8443 which is what I saw locally and thought I configured something wrong (so tried running on Jenkins but get the same failure there).
Perhaps ^ problem with Tomcat automatically starting for some tests is related to the https://github.com/jakartaee/security/pull/284 change for 3.0.1.
any idea why Apache Tomcat might be used for some tests by default?
Yes, two tests use an OpenID Connect Server to test against, and this just happens to be a Tomcat application. It's similar to how e.g. the Jakarta Mail TCK starts up a Mail Server to test against.