Quarkus 3.9.4 "uber jar" builds that use quarkus-resteasy-reactive fails to start (multi-release jar)

[snazy]

Describe the bug

The exception we get is:

Exception in thread "main" java.lang.ExceptionInInitializerError
	at java.base/jdk.internal.misc.Unsafe.ensureClassInitialized0(Native Method)
	at java.base/jdk.internal.misc.Unsafe.ensureClassInitialized(Unsafe.java:1160)
	at java.base/jdk.internal.reflect.MethodHandleAccessorFactory.ensureClassInitialized(MethodHandleAccessorFactory.java:300)
	at java.base/jdk.internal.reflect.MethodHandleAccessorFactory.newConstructorAccessor(MethodHandleAccessorFactory.java:103)
	at java.base/jdk.internal.reflect.ReflectionFactory.newConstructorAccessor(ReflectionFactory.java:200)
	at java.base/java.lang.reflect.Constructor.acquireConstructorAccessor(Constructor.java:549)
	at java.base/java.lang.reflect.Constructor.newInstanceWithCaller(Constructor.java:499)
	at java.base/java.lang.reflect.Constructor.newInstance(Constructor.java:486)
	at io.quarkus.runtime.Quarkus.run(Quarkus.java:70)
	at io.quarkus.runtime.Quarkus.run(Quarkus.java:44)
	at io.quarkus.runtime.Quarkus.run(Quarkus.java:124)
	at io.quarkus.runner.GeneratedMain.main(Unknown Source)
Caused by: java.lang.RuntimeException: Failed to start quarkus
	at io.quarkus.runner.ApplicationImpl.<clinit>(Unknown Source)
	... 12 more
Caused by: java.lang.NoSuchMethodError: 'void org.projectnessie.api.v2.params.AbstractParams.__quarkus_init_converter__maxRecords(org.jboss.resteasy.reactive.server.core.Deployment)'
	at io.quarkus.rest.runtime.__QuarkusInit.init(Unknown Source)
	at io.quarkus.resteasy.reactive.server.runtime.ResteasyReactiveRecorder.createDeployment(ResteasyReactiveRecorder.java:155)
	at io.quarkus.deployment.steps.ResteasyReactiveProcessor$setupDeployment713137389.deploy_6(Unknown Source)
	at io.quarkus.deployment.steps.ResteasyReactiveProcessor$setupDeployment713137389.deploy(Unknown Source)
	... 13 more

Reproducer (as in commit https://github.com/projectnessie/nessie/commit/7219ea40e881f12c668b81ceac986446176394f9, we might revert the resteasy-reactive change when we do a release):

1. git clone https://github.com/projectnessie/nessie.git
2. cd nessie/
3. ./gradlew :nessie-quarkus:quarkusBuild
4. java -jar servers/quarkus-server/build/quarkus-app/quarkus-run.jar --> Works (not an uber-jar)
5. ./gradlew :nessie-quarkus:quarkusBuild -Puber-jar
6. java -jar servers/quarkus-server/build/nessie-quarkus-0.80.1-SNAPSHOT-runner.jar --> Fails

The change that introduced resteasy-reactive was this PR.

(Reference in our project: https://github.com/projectnessie/nessie/issues/8390)

/@geoand

Expected behavior

No response

Actual behavior

No response

How to Reproduce?

No response

Output of uname -a or ver

No response

Output of java -version

No response

Quarkus version or git rev

No response

Build tool (ie. output of mvnw --version or gradlew --version)

No response

Additional information

No response

[quarkus-bot[bot]]

/cc @FroMage (resteasy-reactive), @geoand (resteasy-reactive), @stuartwdouglas (resteasy-reactive)

[snazy]

Some more info: AbstractParams is an abstract base class link - it's used for a bunch of parameters containers in REST requests. For example this one is referenced here

[snazy]

Hm - if I unzip the uber-jar, I see:

javap org/projectnessie/api/v2/params/AbstractParams.class
Compiled from "AbstractParams.java"
public abstract class org.projectnessie.api.v2.params.AbstractParams<IMPL extends org.projectnessie.api.v2.params.AbstractParams<IMPL>> {
  protected org.projectnessie.api.v2.params.AbstractParams();
  protected org.projectnessie.api.v2.params.AbstractParams(java.lang.Integer, java.lang.String);
  public java.lang.Integer maxRecords();
  public java.lang.String pageToken();
  public abstract IMPL forNextPage(java.lang.String);
  public void __quarkus_rest_inject(org.jboss.resteasy.reactive.server.injection.ResteasyReactiveInjectionContext);
  public static void __quarkus_init_converter__maxRecords(org.jboss.resteasy.reactive.server.core.Deployment);
  public static void __quarkus_init_converter__pageToken(org.jboss.resteasy.reactive.server.core.Deployment);
}

[snazy]

Ah! Some hint! There is another AbstractParams.class here: META-INF/versions/11/org/projectnessie/api/v1/params/AbstractParams.class:

javap META-INF/versions/11/org/projectnessie/api/v1/params/AbstractParams.class
Compiled from "AbstractParams.java"
public abstract class org.projectnessie.api.v1.params.AbstractParams<IMPL extends org.projectnessie.api.v1.params.AbstractParams<IMPL>> {
  protected org.projectnessie.api.v1.params.AbstractParams();
  protected org.projectnessie.api.v1.params.AbstractParams(java.lang.Integer, java.lang.String);
  public java.lang.Integer maxRecords();
  public java.lang.String pageToken();
  public abstract IMPL forNextPage(java.lang.String);
}

[snazy]

Yea - the MR-jar is probably the cause here. What I did:

1. mkdir zz ; cd zz
2. unzip ../nessie-quarkus-0.80.1-SNAPSHOT-runner.jar
3. rm -rf META-INF/versions/11/org/projectnessie
4. zip -0r ../foo.jar *
5. java -jar ../foo.jar

But that then runs into this exception (I suspect that's a consequence of the removed MR classes):

Exception in thread "main" java.lang.ExceptionInInitializerError
	at java.base/jdk.internal.misc.Unsafe.ensureClassInitialized0(Native Method)
	at java.base/jdk.internal.misc.Unsafe.ensureClassInitialized(Unsafe.java:1160)
	at java.base/jdk.internal.reflect.MethodHandleAccessorFactory.ensureClassInitialized(MethodHandleAccessorFactory.java:300)
	at java.base/jdk.internal.reflect.MethodHandleAccessorFactory.newConstructorAccessor(MethodHandleAccessorFactory.java:103)
	at java.base/jdk.internal.reflect.ReflectionFactory.newConstructorAccessor(ReflectionFactory.java:200)
	at java.base/java.lang.reflect.Constructor.acquireConstructorAccessor(Constructor.java:549)
	at java.base/java.lang.reflect.Constructor.newInstanceWithCaller(Constructor.java:499)
	at java.base/java.lang.reflect.Constructor.newInstance(Constructor.java:486)
	at io.quarkus.runtime.Quarkus.run(Quarkus.java:70)
	at io.quarkus.runtime.Quarkus.run(Quarkus.java:44)
	at io.quarkus.runtime.Quarkus.run(Quarkus.java:124)
	at io.quarkus.runner.GeneratedMain.main(Unknown Source)
Caused by: java.lang.RuntimeException: Failed to start quarkus
	at io.quarkus.runner.ApplicationImpl.<clinit>(Unknown Source)
	... 12 more
Caused by: jakarta.validation.ConstraintDeclarationException: HV000151: A method overriding another method must not redefine the parameter constraint configuration, but method RestNamespaceResource#deleteNamespace(NamespaceParams) redefines the configuration of HttpNamespaceApi#deleteNamespace(NamespaceParams).
	at org.hibernate.validator.internal.metadata.aggregated.rule.OverridingMethodMustNotAlterParameterConstraints.apply(OverridingMethodMustNotAlterParameterConstraints.java:24)
	at org.hibernate.validator.internal.metadata.aggregated.ExecutableMetaData$Builder.assertCorrectnessOfConfiguration(ExecutableMetaData.java:462)
	at org.hibernate.validator.internal.metadata.aggregated.ExecutableMetaData$Builder.build(ExecutableMetaData.java:380)
	at org.hibernate.validator.internal.metadata.aggregated.BeanMetaDataBuilder$BuilderDelegate.build(BeanMetaDataBuilder.java:260)
	at org.hibernate.validator.internal.metadata.aggregated.BeanMetaDataBuilder.build(BeanMetaDataBuilder.java:133)
	at org.hibernate.validator.internal.metadata.PredefinedScopeBeanMetaDataManager.createBeanMetaData(PredefinedScopeBeanMetaDataManager.java:155)
	at org.hibernate.validator.internal.metadata.PredefinedScopeBeanMetaDataManager.<init>(PredefinedScopeBeanMetaDataManager.java:100)
	at org.hibernate.validator.internal.engine.PredefinedScopeValidatorFactoryImpl.<init>(PredefinedScopeValidatorFactoryImpl.java:206)
	at org.hibernate.validator.PredefinedScopeHibernateValidator.buildValidatorFactory(PredefinedScopeHibernateValidator.java:42)
	at org.hibernate.validator.internal.engine.AbstractConfigurationImpl.buildValidatorFactory(AbstractConfigurationImpl.java:435)
	at io.quarkus.hibernate.validator.runtime.HibernateValidatorRecorder$2.created(HibernateValidatorRecorder.java:180)
	at io.quarkus.arc.runtime.ArcRecorder.initBeanContainer(ArcRecorder.java:79)
	at io.quarkus.deployment.steps.ArcProcessor$notifyBeanContainerListeners1304312071.deploy_0(Unknown Source)
	at io.quarkus.deployment.steps.ArcProcessor$notifyBeanContainerListeners1304312071.deploy(Unknown Source)
	... 13 more

[geoand]

Thanks for the investigation!

This is pretty weird indeed, I'll have to take a close look soon.

[snazy]

What I don't really understand is why the "normal" jar works :shrug:

BTW: We'll push a workaround to Nessie soon.

[snazy]

Um - so the workaround makes things actually worse. EDIT: There's something odd with dependencies w/ classifiers - but not related to this issue.

[geoand]

I can indeed reproduce the problem but before going further I would like to know how the file for the MR Jar is being created.

[snazy]

TL;DR our nessie-model.jar contains classes with both javax.* and jakarta.* annotations. Since the jakarta.* annotations require a Java 11 runtime (and we still have to support Java 8 for clients), we produce a MR-jar: The "main" classes only have the javax.* annotations, while the MR-jar has both javax.* and jakarta.* annotations. The jakarta-annotations are "stripped" via a Gradle plugin.

The jar's basically configured in this plugin here. The build script refers to the plugin here.

Note: the current state of the Nessie build scripts have a (really hacky) workaround for this (https://github.com/projectnessie/nessie/pull/8394), which we would later remove.

[snazy]

You can create the MR-jar using ./gradlew :nessie-model:jar - it's then here: api/model/build/libs/nessie-model-0.80.1-SNAPSHOT.jar

[geoand]

Thanks for the info.

So in short I don't see how we can solve this one... The problem is that when we transform classes (as we do for bean params) we only write the result to the canonical .class file, not any of the JDK version specific ones. Actually, it's even worse than that as the transformation is only done on the non JDK specific version of the .class file.

[snazy]

Would it be a simple thing to transform all classes under META-INF/versions as well? Or does the transformation need some complex context to do its job?

[snazy]

BTW: I wonder why this only happens with a Quarkus uber-jar but not a fast-jar.

[geoand]

Would it be a simple thing to transform all classes under META-INF/versions as well?

Theoretically yes, this could be done as well but the context is not currently setup to handle multiple versions of the same class.

[geoand]

BTW: I wonder why this only happens with a Quarkus uber-jar but not a fast-jar.

I am pretty sure that the JDK specific version is not getting loaded at all in this case for the simple reason that our ClassLoader does not support it (AFAIR)

[gastaldi]

Worth mentioning that MultiRelease JARs are not properly read when we use new JarFile(file), because it uses the base version (which is 8) instead of the JarFile.runtimeVersion().

The io.smallrye.common.io.jar.JarFiles already handles that, so maybe it would be nice to use that to create JarFile instances in our codebase.

[geoand]

Sure, but that the least of our problems with it comes to this :)

[FroMage]

In order to fix this, we'd have to be able to make the Quarkus build work with MR jars, which I don't think it does ATM.

I don't think Jandex will index all versions of a class, I also don't think our bytecode transformer architecture works with them. We'd have to transform all versions, and produce them in the proper versioned places.

In other words, this is a much bigger issue than this special case, I'm afraid.

[snazy]

From our perspective, we have a workaround (basically: have a non-MR-jar) - so we're fine.

Generally speaking, I think it's fine to say: MR-Jar specialties aren't supported. It's IMHO extremely tricky - just thinking about having an endpoint for Java 22+ but not for Java 17 (so "completely different" per Java version) - I don't think that's feasible. Quarkus would have to support MR-jars at build time and at run time. Sounds extremely complex for little win?

[FroMage]

It would be a lot of work, for sure.